Determining speaker changes in audio input

ABSTRACT

Intelligent assistant systems, methods and computing devices are disclosed for identifying a speaker change. A method comprises receiving audio input comprising a speech fragment. A first voice model is trained with a first sub-fragment from the speech fragment. A second voice model is trained with a second sub-fragment from the speech fragment. The first sub-fragment is analyzed with the second voice model to yield a first confidence value. The second sub-fragment is analyzed with the first voice model to yield a second confidence value. Based at least on the first and second confidence values, the method determines if a speaker of the first sub-fragment is the speaker of the second sub-fragment.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/459,020 filed Feb. 14, 2017, and to U.S. Provisional PatentApplication No. 62/482,165 filed Apr. 5, 2017, the entirety of which arehereby incorporated herein by reference.

BACKGROUND

Interacting with computing systems via natural interactions, such as oneor more of voice recognition, text, gesture recognition, motiondetection, gaze detection, etc., enables natural user interfaceexperiences. As the volume of digital information and the number ofcomputing devices increases, managing such natural user interactioninterfaces to provide positive user experiences can prove challenging.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example environment with an intelligent assistant systemin the form of an all-in-one computing device according to an example ofthe present disclosure.

FIG. 2 schematically shows an example logical architecture forimplementing an intelligent assistant system according to an example ofthe present disclosure.

FIG. 3 schematically shows a speech recognition program that may beutilized by a voice listener according to an example of the presentdisclosure.

FIG. 4 shows an intent template according to an example of the presentdisclosure.

FIG. 5 schematically shows a parser and an intent handler processing aportion of a conversation according to an example of the presentdisclosure.

FIGS. 6A and 6B show a method for addressing missing and/or unresolvedinformation in an intent template according to examples of the presentdisclosure.

FIG. 7 schematically shows an entity tracker that may determine anidentity, position, and/or current status of one or more entitiesaccording to examples of the present disclosure.

FIG. 8 schematically shows an entity tracker receiving and interpretingsensor data over multiple time frames according to examples of thepresent disclosure.

FIG. 9 schematically shows an example of sensor confidence decay overtime via an entity tracker according to an example of the presentdisclosure.

FIG. 10 schematically shows an example of using a trained voicerecognition engine to recognize a person's speech according to examplesof the present disclosure.

FIG. 11 schematically shows an example of using sub-fragments of audiodata to train voice models according to examples of the presentdisclosure.

FIG. 12 schematically shows an example of cross-evaluating sub-fragmentsof audio data to determine whether the active speaker has changedaccording to examples of the present disclosure.

FIG. 13 shows a schematic illustration of a cost function according toexamples of the present disclosure.

FIG. 14 shows a flowchart of a method for determining a cost ofnotification and a cost of non-notification according to examples of thepresent disclosure.

FIG. 15 shows a schematic representation of a threshold updating moduleaccording to examples of the present disclosure.

FIG. 16 shows a flowchart of a method for updating threshold dataaccording to examples of the present disclosure.

FIG. 17 schematically shows an all-in-one computing device thatimplements an intelligent assistant system according to examples of thepresent disclosure.

FIG. 18 schematically shows an example implementation in which one ormore remote services perform functionality of the intelligent assistantsystem according to examples of the present disclosure.

FIG. 19 schematically shows another example implementation in which oneor more remote services perform functionality of intelligent assistantsystem according to examples of the present disclosure.

FIG. 20 schematically shows another example implementation in which oneor more remote services utilizes a device selector according to examplesof the present disclosure.

FIG. 21 schematically shows an example implementation in which one ormore functions of the intelligent assistant system are activated upondetection of one or more spoken keywords.

FIG. 22 schematically shows an example implementation of a multi-deviceenvironment in which sensor(s) and output device(s) are selected inresponse to voice activation according to examples of the presentdisclosure.

FIG. 23 schematically shows a computing system according to examples ofthe present disclosure.

FIG. 24 schematically shows a portion of a speech fragment divided intosub-fragments of different lengths according to examples of the presentdisclosure.

FIGS. 25A and 25B are a flow chart of a method for identifying a speakerchange according to examples of the present disclosure.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

Intelligent digital assistant systems, methods and computing devices aredisclosed for identifying a speaker change. In one example, a methodcomprises receiving audio input comprising a speech fragment. A firstvoice model is trained with a first sub-fragment from the speechfragment. A second voice model is trained with a second sub-fragmentfrom the speech fragment. The first sub-fragment is analyzed with thesecond voice model to yield a first confidence value. The secondsub-fragment is analyzed with the first voice model to yield a secondconfidence value. Based at least on the first and second confidencevalues, the method determines if a speaker of the first sub-fragment isthe speaker of the second sub-fragment.

DETAILED DESCRIPTION Overview

The present disclosure relates generally to systems, methods and logicalconstructs for providing intelligent assistance to users. In someexamples, a variety of sensor data may be utilized to intelligentlydetermine the content and/or timing of messages communicated to usersand/or the performance of actions. In some examples natural languageinputs, such as user commands and other utterances, may be received andprocessed. In some examples, a natural language input may be parsed andanalyzed to generate an indication of one or more user intentionsassociated with the input. In some examples, data from one or moresensors also may be utilized to process the natural language inputsand/or user intentions. Such data may be processed to generate identity,location/position, status/activity, and/or other information related toone or more entities within range of a sensor. Statistical probabilitiesbased on current and past data may be utilized to generate confidencevalues associated with entity information.

User intentions may be processed to at least partially resolvelinguistic, semantic and/or other ambiguities. Using the resultingclarified intention, a commitment for carrying out the intention may begenerated and either executed or stored. In determining whether and whento execute a commitment, one or more factors may be examined. In someexamples, factors related to the importance of the commitment to aparticular user, the receptivity of the user to receiving input, and/orthe user's current context may be estimated. Machine learning techniquesmay be applied to such factors and other data to learn and makepredictions from such information.

Following are descriptions of example implementations and use cases ofan intelligent assistant system for processing natural language inputs.Additional details of various aspects of the system are provided below.

Example Environment

FIG. 1 shows an example of a living room 4 with one example of anintelligent assistant system in the form of an all-in-one computingdevice 10. As described in more detail below, in some examples computingdevice 10 may be configured to receive and process natural languageinputs. A user may utilize the intelligent assistant system for myriadfunctions. For example, the user may provide natural language input toask the intelligent assistant system to perform a variety of tasks, suchas transferring an instance of a computer game from one device toanother. In another example, such a transfer may be performedprogrammatically without input from the user. For example, computingdevice 10 may utilize sensor data, such as audio and/or video data, todetect when the user moves to another room and is looking at or“engaged” with another device. Using this data, computing device 10 mayautomatically transfer the instance of the computer game to the otherdevice.

The user may ask the system for information about a wide range oftopics, such as the weather, personal calendar events, movie show times,etc. In some examples, the intelligent assistant system also may beconfigured to control elements in the living room 4, such as atelevision 6, speakers 8 of a music system, a gas fireplace 14, ormotorized curtains 16.

The intelligent assistant system also may be utilized to receive andstore messages and/or reminders to be delivered at an appropriate futuretime. Using data received from sensors, the intelligent assistant systemmay track and/or communicate with one or more users or other entities.

In some examples, the computing device 10 may be operatively connectedwith one or more other computing devices using a wired connection, ormay employ a wireless connection via Wi-Fi, Bluetooth, or any othersuitable wireless communication protocol. For example, the computingdevice 10 may be communicatively coupled to one or more other computingdevices via a network. The network may take the form of a local areanetwork (LAN), wide area network (WAN), wired network, wireless network,personal area network, or a combination thereof, and may include theInternet. Additional details regarding components and computing aspectsof the computing device 10 are described in more detail below withreference to FIG. 23.

It will be appreciated that the computing device 10 of FIG. 1 is merelyone example implementation of the intelligent assistant system of thepresent disclosure. Additional example implementations across two ormore devices are illustrated in FIGS. 17-22 and described in more detailbelow.

Architecture

FIG. 2 shows an example logical architecture for implementing anintelligent assistant system 20 capable of recognizing and responding tonatural language inputs according to examples of the present disclosure.As described in more detail below, in various examples the system 20 maybe implemented in a single computing device, across two or more devices,in a cloud-supported network, and in combinations of the foregoing.

In this example the intelligent assistant system 20 includes at leastone sensor 22, an entity tracker 100, a voice listener 30, a parser 40,an intent handler 50, a commitment engine 60, and at least one outputdevice 70. In some examples the sensors 22 may include one or moremicrophones 24, visible light cameras 26, infrared cameras 27, andconnectivity devices 28, such as Wi-Fi or Bluetooth modules. In someexamples sensor(s) 22 may comprise stereoscopic and/or depth cameras,head trackers, eye trackers, accelerometers, gyroscopes, gaze detectiondevices, electric-field sensing componentry, GPS or other locationtracking devices, temperature sensors, device state sensors, and/or anyother suitable sensor.

The entity tracker 100 is configured to detect entities and theiractivities, including people, animals, or other living things, as wellas non-living objects. Entity tracker 100 includes an entity identifier104 that is configured to recognize individual users and/or non-livingobjects. Voice listener 30 receives audio data and utilizes speechrecognition functionality to translate spoken utterances into text.Voice listener also may assign confidence value(s) to the translatedtext, and may perform speaker recognition to determine an identity ofthe person speaking, as well as assign probabilities to the accuracy ofsuch identifications. Parser 40 analyzes text and confidence valuesreceived from voice listener 30 to derive user intentions and generatecorresponding machine-executable language.

Intent handler 50 receives the machine-executable language representinguser intentions from the parser 40, and resolves missing and ambiguousinformation to generate commitments. Commitment engine 60 storescommitments from the intent handler 50. At a contextually appropriatetime, the commitment engine may deliver one or more messages and/orexecute one or more actions that are associated with one or morecommitments. Commitment engine 60 may store messages in a message queue62 or cause one or more output devices 70 to generate output. The outputdevices 70 may comprise one or more of speaker(s) 72, video display(s)74, indicator light(s) 76, haptic device(s) 78, and/or other suitableoutput devices. In other examples, output devices 70 may comprise one ormore other devices or systems, such as home lighting, thermostats, mediaprograms, door locks, etc., that may be controlled via actions executedby the commitment engine 60.

In different examples the voice listener 30, parser 40, intent handler50, commitment engine 60, and/or entity tracker 100 may be embodied insoftware that is stored in memory and executed by one or more processorsof a computing device. Additional details regarding the components andcomputing aspects of computing devices that may store and execute thesemodules are described in more detail below with reference to FIG. 23.

Additional descriptions of the components of intelligent assistantsystem 20 will now be provided. In some examples, voice listener 30 mayreceive audio data from the surrounding environment. In some examples,such as in computing device 10 of FIG. 1, the voice listener 30 maycomprise a software module that is embodied in a standalone device thatcomprises one or more microphones. In other examples, the voice listener30 software module may be stored in memory of a computing device that islocated remotely from the user's environment, such as in a cloud-basedservice. In some examples, additional data from one or more othersensors may be received and utilized by the voice listener 30 inperforming its functions that are described in more detail below.

The voice listener 30 may comprise speech recognition functionality thattranslates audio data of spoken utterances into text. As described inmore detail below, the voice listener 30 also may assign a confidencevalue to one or more portions of translated text, such as individualspeech components, words, phrases, etc. With reference now to FIG. 3, insome examples the voice listener 30 may comprise a speech recognitionprogram 120 stored in non-volatile storage 122 of a computing device124. The speech recognition program 120 may be loaded into memory 126and executed by a processor 128 of computing device 124 to perform oneor more of the methods and processes for speech recognition described inmore detail below.

Audio input 130 in the form of natural language speech may be capturedby microphone 24 and processed by audio processor 134 to create audiodata. Audio data from the audio processor 134 may be transformed byfeature extractor 136 into data for processing by a speech recognitionengine 140 of the speech recognition program 120. In some examples,feature extractor 136 may identify portions of the audio data over atime interval that contain speech for processing. Feature extractor 136may extract feature vectors 142 from such portions of the data, with afeature vector representing the qualities of a spoken utterance withinthe time interval of a given portion. A matrix of multiple featurevectors 142 may be provided to the speech recognition engine 140 forfurther processing.

Feature extractor 136 may utilize any suitable dimensionality reductiontechniques to process the audio data and generate feature vectors 142.Example techniques include using mel-frequency cepstral coefficients(MFCCs), linear discriminant analysis, deep neural network techniques,etc.

The speech recognition engine 140 may compare the feature vectors 142generated by feature extractor 136 with acoustic models for speechsounds (e.g., speech components). Examples of speech components mayinclude phonemes, phones, diphones, triphones, etc. In some examples,the speech recognition engine 140 may comprise an acousticrepresentation generator 144 (e.g., acoustic modeler) that evaluates thesimilarity of a spoken utterance represented by one or more featurevectors 142 to acoustic models of language sounds. The acoustic modelsmay comprise data that matches pronunciations of speech components, suchas phonemes, to particular words and/or phrases.

The speech recognition engine 140 also may compare the feature vectorsand other audio data with sequences of sounds to identify words and/orphrases that match the spoken sounds of the audio data. The speechrecognition program 120 may comprise a language representation generator146 (e.g., language modeler) that may utilize language models toevaluate the likelihood that a particular word would be included in aphrase (which in some cases may comprise a sentence) at a particularlocation. For purposes of the present disclosure, a phrase may includetwo or more words that may or may not be considered a complete sentence.

In some examples, the speech recognition engine 140 may utilize HiddenMarkov models (HMMs) to match feature vectors 142 with phonemes and/orother speech components. An HMM outputs sequences of n-dimensionalvectors, where n is an integer such as 10. Sequences may be generated ata given frequency, such as one sequence every 10 milliseconds.

Each state of an HMM may comprise a statistical distribution that is amixture of diagonal covariance Gaussians, which may indicate alikelihood for each observed vector. Each phoneme or word may have adifferent output distribution. Individual HMMs for separate phonemes andwords may be combined to create an HMM for a sequence of phonemes orwords.

Context dependency for phonemes may be provided by different states ofan HMM. Such context-dependent HMM states may be associated with amodel, such as a Gaussian mixture model (GMM). In some examples,transitions between states may be assigned probabilities that correspondto a likelihood that a current state may be reached from a previousstate. Different paths between states of the HMM may represent inputtedsounds, with the different paths representing multiple possible textmatches for the same sound.

Using the feature extractor 136 and speech recognition engine 140, thespeech recognition program 120 may process feature vectors 142 and otherspeech recognition data 148 to generate recognized text 66. In otherexamples, any suitable techniques for matching feature vectors 142 tophonemes and/or other speech components may be utilized.

In some examples, the speech recognition program 120 may determineestimated confidence values 152 for one or more portions of the speechrecognition data 148, such as individual speech components, words andphrases. An estimated confidence value 152 may define a statisticallikelihood that the corresponding recognized text is accurate. Asdescribed in more detail below, the parser 40 of intelligent assistantsystem 20 may utilize such confidence values 152 in processingrecognized text and determining a user's intent.

In different examples, confidence values 152 may be determined byutilizing one or more statistical analysis methods, machine learningtechniques, empirically-derived data, and combinations of the foregoing.In some examples, the speech recognition program 120 may utilize one ormore probabilistic models to analyze portions of the speech recognitiondata 148, one or more results extracted from the speech recognitionanalysis pipeline, and/or estimated confidence values 152 associatedwith such portions. For example, GMMs may be utilized to analyzeportions of the speech recognition data 148 and corresponding results.It will be appreciated that any other suitable machine learningtechniques, such as various supervised learning and unsupervisedlearning approaches, may be utilized to analyze the speech recognitiondata 148.

It will be appreciated that the foregoing descriptions of speechrecognition techniques are merely examples, and that any suitable speechrecognition technologies and processes may be utilized and arecontemplated within the scope of the present disclosure.

With reference again to FIG. 2, in some examples the voice listener 30may receive context information including associated confidence valuesfrom entity tracker 100. As described in more detail below, entitytracker 100 may determine an identity, position, and/or current statusof one or more entities within range of one or more sensors, and mayoutput such information to one or more other modules, such as voicelistener 30, commitment engine 60, etc. In some examples, entity tracker100 may interpret and evaluate sensor data received from one or moresensors, and may output context information based on the sensor data.Context information may include the entity tracker's guesses/predictionsas to the identity, position, and/or status of one or more detectedentities based on received sensor data. In some examples, theguesses/predictions may additionally include a confidence value definingthe statistical likelihood that the information is accurate.

Additional details regarding components and computing aspects that maybe used to implement voice listener 30 are described in more detailbelow with respect to FIG. 23.

With continued reference to FIG. 2, the voice listener 30 may sendrecognized text and corresponding confidence values to the parser 40. Asdescribed in more detail below, the parser 40 analyzes the text andconfidence values to determine an intent of the user in speaking thereceived utterance. The parser 40 may translate the natural languagetext received from the voice listener 30 into code or data structuresthat represent a user's intention underlying the natural language.

In some examples, a user's intention may correspond to a command to beexecuted immediately, such as the utterance “Play song A by artist B” (a“Play music” intent). In some examples, an intent may be characterizedas a commitment to execute an action upon the occurrence of a trigger,hereinafter referred to as an “add commitment” intent. For example, theutterance “When Bob gets home remind him to take out the trash” is anadd commitment intent. In this example, the trigger is Bob arrivinghome, and the action is to remind him to take out the trash. Anotherexample of an add commitment intent may be the utterance “When Keith isnear the oven, alert me.” In this example, the commitment of this addcommitment intent comprises a trigger (Keith is near the oven) and anaction (alert me) to be executed when the trigger is detected.Additional descriptions and examples of commitments are provided below.

In some examples the parser 40 may utilize a plurality of intenttemplates that each contain a plurality of slots that may be filled withwords or terms received from the voice listener 30, or with words orterms that are based on other words received from the voice listener. Insome examples where one or more slots are not filled, the parser 40 mayfill these slots by examining a semantic meaning of one or more otherwords. For example, the intelligent assistant system 20 may tell a user,“You have 15 emails.” The user may respond with an utterance, “OK, I'llgo through them when I'm in the car.” In response to the user'sutterance, the parser 40 may fill a “commitment type” slot with the type“reminder”, even though the word “reminder” itself was not in the user'sutterance.

Taken together, the plurality of slots of an intent template define orotherwise characterize the intent of the user in speaking an utterance.In various different examples, the slots may comprise an action slot, atrigger slot, a commitment slot, a subject slot, a content slot, anidentity slot, and various other types of slots. In some examples, eachslot may embody one of three states: (1) missing information, (2)information present with unresolved ambiguity, and (3) informationpresent with any ambiguity resolved.

In some examples, one or more slots may be optional slots that need notbe filled. For example, in one scenario two slots may represent optionalinformation, while in another scenario the same two slots may representrequired information. For example, the utterance “Play music” may beunderstood as a command that music should be played out of the devicebeing used for this conversation. In this manner, the system infersinformation regarding the user's intention (to play music via the devicebeing used for the conversation) without requiring the user toexplicitly state this information. In a different example, the utterance“Whenever it's Eve's birthday, play Happy Birthday” will require theuser to specify the device to use, since the play music action isscheduled to be performed some time in the future whenever the specifiedcondition is met.

One example of an intent template is a commitment intent template thatcorresponds to an add commitment intent. With reference now to FIG. 4,one example of a commitment intent template 200 is illustrated. In thisexample, the parser 40 may receive text phrase 210 from the voicelistener 30 that reads “When Keith is near the oven alert me.” Thephrase “When Keith is near the oven” may be identified as a trigger 214.The phrase “alert me” may be identified as an action 218 that is to becarried out when the trigger is detected. As described in more detailbelow, in some examples the parser 40 may translate this text phrase 210into machine-executable language that is passed to the intent handler 30for further processing.

As noted above, the parser 40 may receive accuracy confidence valuesfrom the voice listener 30 that denote a likelihood that correspondingtext is accurate. In some examples and as described in more detailbelow, the intent handler 50 also may receive entity confidence valuesthat are associated with entity information. In some examples, suchentity confidence values and other context information may be receivedvia the entity tracker 100.

In the present example, the word “me” in phrase 210 fills a subject slot222. In this example, the subject slot 222 corresponds to the person orother entity to be alerted when the trigger is detected. The word “me”may be received by the parser 40 with context information thatassociates this word to a particular person named Joe, and with anentity confidence value, such as 90%, that denotes a level of certaintythat “me” is the person “Joe.”

In some examples, the intended meaning of one or more words in an intenttemplate may not be readily apparent. For example, in phrase 210 themeaning of the word “near” may be ambiguous, as “near” is a relativeterm. A variety of contextual factors may influence the intended meaningof “near” and the corresponding distance contemplated in this phrase.For example, where “Keith” is an infant, the intended meaning of “near”may be based on important safety concerns of the user speaking thephrase. Where “Keith” is the husband of the user, the intended meaningof “near” may be influenced less by safety concerns and more byconvenience factors, which may lead to an associated distance that isdifferent from the case where “Keith” is an infant. In another example,the distance intended to be conveyed in the phrase “near the oven” islikely different from the distance intended to be conveyed in the phrase“near the Statue of Liberty.”

Accordingly, one or more words in an intent template may be ambiguous aspassed to the intent handler 50. As described in more detail below, theintent handler 50 may utilize a plurality of techniques to resolveambiguities and to fill in slots with missing information in an intenttemplate.

In another example, the parser 40 may receive the text phrase “Playmusic with Fred” from the voice listener 30. In some examples, thephrase “Play music” is often interpreted to mean that a user wants toplay digital music files via a media player. However, the use of thephrase “with Fred” following “Play music” is unusual, as peopletypically would not use this phrasing when their intent is to play musicvia a media player. The parser 40 may recognize this ambiguity and maygenerate a list of N-best intent templates that it determines are thestatistically most probable intent templates corresponding to the user'sactual intent. In some examples, the intent handler 50 may useadditional context information to select an intent template from thelist of N-best intent templates.

In another example, the text phrase received from the voice listener 30may be the single word “Play.” For example, the word or words spoken bythe user after “Play” may have been unintelligible to the voice listenerfor one or more reasons (such as loud noises in the background). In thisexample, the parser 40 may predict that the user's intent is to playdigital music, but in the corresponding intent template the content slotrepresenting what music to play is empty. In this example, the parser 40may send a “Play music” intent template to the intent handler 50 forfurther processing and resolution of this ambiguity, as described inmore detail below.

In some examples, the parser 40 may analyze received text to form adecision tree of the user's intent. In some examples, the parser 40 maygenerate If-Then statements (or rules) from the received text. EachIf-Then statement may comprise a corresponding trigger and an action.Whenever the conditions of the trigger are satisfied, the action isperformed. The resulting If-Then statements can perform a wide varietyof tasks, such as home security (“text me if the motion detector in theback yard is activated”), home automation (“turn on the fireplace when Iarrive home”), personal organization (“collect my email receipts forcharitable donations into a spreadsheet”), health-related tasks (“remindme to eat protein if I run more than 7 miles”), and many others.

In some examples, triggers and actions may be drawn from a range ofchannels that may be activated by a user. These channels may representdifferent entities and services, including devices (such as smart phoneoperating systems, connected home components such as smart lightswitches, etc.), knowledge sources (such as entertainment websites,email providers, etc.), and the like. Each channel may expose a set offunctions for both the trigger and the action.

For example, If-Then statements may take the form of “IF [Input(s)] arerecognized, THEN perform [Action(s)]”. For example, the received phrase“When Oz is in the kitchen, tell him to take out the garbage” may betranslated to the following If-Then statement: “IF the person Oz isdetermined to be in the kitchen, THEN broadcast a message to the personOz to take out the garbage.” In some examples, the parser 40 maydetermine that a user intends to establish a recurring a message oraction based on parsing a received utterance. For example, in the phrase“When Oz is in the kitchen, tell him to take out the garbage,” the word“when” may be interpreted by the parser 40 to designate that thecorresponding action should be performed each time the condition is met(i.e., each time Oz is in the kitchen, tell him to take out thegarbage). In another example, in the phrase “If Oz is in the kitchen,tell him to take out the garbage,” the word “if” may be interpreted todesignate that the corresponding action should be performed one timeonly (i.e., the next time Oz is in the kitchen, tell him to take out thegarbage).

In some examples and as noted above, these If-Then statements may begenerated probabilistically. In this manner and for a given string oftext, the parser 40 may generate a plurality of N-best candidates ofIf-Then statements that may correspond to the user's utterance.

In some examples of parsing If-Then rules, the parser 40 may utilize ameaning representation that comprises an abstract syntax tree (AST) in avery simple language. For example, each root node may expand into a“trigger” and “action” pair. These nodes in turn expand into a set ofsupported triggers and actions. These trees may be modeled as a nearlycontext-free grammar that generates If-Then tasks. Additionaldescription of semantic parsers for If-Then statements is provided inthe following publications: “Language to Code: Learning Semantic Parsersfor If-This-Then-That Recipes”, authored by Chris Quirk, Raymond Mooney,and Michel Galley, Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics, pages 878-888, Beijing,China, Jul. 26-31, 2015, the entirety of which is incorporated herein byreference; and “Improved Semantic Parsers For If-Then Statements”authored by I. Belagy and Chris Quirk, Proceedings of the 54th AnnualMeeting of the Association for Computational Linguistics, pages 726-736,Berlin, Germany, Aug. 7-12, 2016, the entirety of which is incorporatedherein by reference. Additional descriptions of techniques for modelingrelation paths in embedding models for knowledge bases and text areprovided in “Compositional Learning of Embeddings for Relation Paths inKnowledge Bases and Text”, authored by Kristina Toutanova, Xi VictoriaLin, Wen-tau Yih, Hoifung Poon, and Chris Quirk, Proceedings of the 54thAnnual Meeting of the Association for Computational Linguistics, pages1434-1444, Berlin, Germany, Aug. 7-12, 2016, the entirety of which isincorporated herein by reference.

In some examples, the parser 40 may use an ensemble of two techniques togenerate If-Then statements and/or derive an intent from the textreceived from the voice listener 30: (1) a recurrent neural network(RNN) architecture in the form of a long short-term memory (LSTM)network, and (2) a logistic regression model. In some examples, a graphlong short term memory (graph LSTM) neural network may be utilized toextract from received text semantic meanings and relationships betweenwords that are inherent to natural language. For example, text may beparsed using a graph LSTM neural network to extract cross-sentence n-aryrelationships using several graph LSTM units arranged according to thesyntactic relations of terms in the segment of text. These syntacticrelationships between words may be tracked in the graph LSTM neuralnetwork to allow artificial intelligence and machine learning techniquesto identify entities and their context within the text and from thegrammatical structure in which they exist.

For example, context that identifies the nouns to which pronouns refer,the adverbs that modify given verbs, the prepositional phrases thataffect a given word, etc., may be incorporated into the various words toenable more accurate searches of the contents of natural languagedocuments. Additional descriptions of and examples of using graph LSTMneural networks to extract semantic meanings and relationships betweenwords are provided in U.S. patent application Ser. No. 15/395,961,entitled GRAPH LONG SHORT TERM MEMORY FOR SYNTACTIC RELATIONSHIPDISCOVERY, filed on Dec. 30, 2016, the entire contents of which areincorporated herein by reference.

In some examples, the parser 40 may receive and process text to graphnodes (e.g., words, phrases, characters, etc.) and edges (e.g.,dependency links between nodes) in individual phrases and acrossboundaries of phrases. In various examples, the graphing may includeidentifying one or more links (e.g., syntactic, semantic, co-reference,discourse, etc.) between nodes in the text. The links can includeintra-phrase and inter-phrase links between nodes. For example, a linkcan represent a relationship between the root of one phrase and the rootof an adjacent phrase. For another example, a link can represent arelationship between two words in a phrase, such as the modifier“Annie's” to the word “lunch.” Additional details regarding graphingnodes and edges in phrases and across boundaries of phrases is disclosedin U.S. patent application Ser. No. 15/173,349, entitled RELATIONEXTRACTION ACROSS SENTENCE BOUNDARIES, filed on Jun. 3, 2016, the entirecontents of which are incorporated herein by reference.

Additional details regarding components and computing aspects that maybe used to implement parser 40 are described in more detail below withrespect to FIG. 23.

As described above, in some examples the parser 40 passes an intenttemplate to the intent handler 50 for further processing. The intenthandler 50 comprises a multi-step pipeline that may resolve ambiguousinformation and/or information that is missing from an intent template.As described in more detail below, the intent handler 50 may utilize aplurality of techniques to resolve ambiguities and fill in missinginformation slots with respect to an intent template. In some examples,the intent handler 50 may utilize domain-specific information anddomain-specific reasoning to resolve ambiguities, complete missinginformation, and otherwise clarify an intent template to more closelycorrespond to the actual intent of the user.

In some examples, the intent handler 50 may glean knowledge regardingthe user's intent by analyzing prior utterances of the user in aconversation history, and may utilize such insights to resolveambiguities and add missing information to an intent template. Once theintent handler 50 has sufficiently clarified ambiguities and completedmissing information, a corresponding commitment may be generated andpassed to the commitment engine 60 for execution.

The intent handler 50 may be configured to process multiple intenttemplates that may comprise a conversation. For purposes of the presentdisclosure and as described in more detail below, a conversation maycomprise a plurality of information and other data related to one ormore exchanges between the user and the intelligent assistant system 20.In different examples, such information and data may comprise wordsand/or phrases spoken by a user, queries presented to the user by theintelligent assistant system 20, sensor data received from one or moresensors, context information such as person and/or identity information,etc.

As described in the use case examples provided below, the intent handler50 may comprise a plurality of resolvers that translate intent templatesand their associated data received from the parser 40 into internal datareferences. To address slots that comprise missing and/or unresolvedinformation in an intent template, the intent handler 50 may utilize theplurality or resolvers in a multi-stage process. In some examples, eachof the resolvers may be specifically programmed to handle issuesassociated with a particular intent template that may be received fromthe parser 40.

Examples of resolvers may include lookup resolvers that translate propernames, aliases, and other identifiers into internal representation data(for example, “Bob” is translated to an internal representation of theperson “Bob”, such as Bob's contact information). Examples of resolversmay include anaphoric resolvers that address expressions having aninterpretation that depends upon an antecedent or postcedent expressionin context (for example, “she” is translated to a slot representing “apersonal identity of the pronoun ‘she’”), and deixis resolvers thataddress words and phrases, such as “here” or “there”, that cannot befully understood without additional contextual information (for example,“there” may translated to a slot representing “where is there?”). Inother examples, many other forms and types of resolvers may be utilized.

With reference now to FIG. 5, one example of the parser 40 and intenthandler 50 processing a portion of a conversation is schematicallyillustrated. In this example, a first phrase 1 is parsed by the parser40 into an intent template 1. The parser 40 provides intent template 1to the intent handler 50, which utilizes a first resolver 1 to resolveambiguities and/or missing information in this intent template. A secondintent template 2 corresponding to a second phrase 2 is received fromthe parser 40. As described in more detail below, the intent handler 50may analyze the intent template 2 along with context information 110 todetermine whether to utilize first resolver 1 or second resolver 2 toresolve the intent template 2. A third intent template 3 based on athird parsed phrase 3 may then be received by the intent handler 50. Theintent handler 50 may utilize a third resolver 3 to resolve intenttemplate 3. Additional details and use case examples of analyzing intenttemplates with resolvers are provided below.

In some examples the intent handler 50 may determine whether two or moreintent templates should be fused or merged together to continue with anexisting conversation path. If the intent handler 50 determines that thetwo or more intent templates should be fused together, then the intenthandler may fuse the data associated with the two or more intenttemplates and continue following the existing conversation path with thefused data. If the intent handler 50 determines that the two or moreintent templates should not be fused together, then a new topic may bestarted using the most recently received intent template.

As described in more detail below, where a slot of an intent templatehas missing information, the intent handler 50 may perform datagathering operations (such as to ask the user to clarify or provideinformation, or try to gather the information in another way) in orderto populate information in the slot. Once each slot containsinformation, the intent handler 50 may determine if the information ineach slot is unambiguous. For information identified as ambiguous, theintent handler 50 may apply one or more of a variety of techniques toresolve the ambiguity.

With reference again to FIG. 2, in some examples the intent handler 50may comprise a mapper 52 that maps one or more system goals to acorresponding user intent(s). Examples of system goals may includeclarifying ambiguities, acquiring additional information from a user,etc. In some examples, mapper 52 may internally rephrase system goals asuser intents or goals. For example, mapper 52 may map information thesystem needs, such as information to resolve an ambiguous intent, to auser intent that the user would have triggered in providing thatinformation. In other words, mapper 52 may map information to the intentthat would have been resolved from an utterance that a user would havespoken in order to generate the intent. In some examples, mapper 52 maymap a system goal to a word or phrase the user would have said togenerate the same outcome.

In some examples, where the system needs information from a user toresolve a user intent, the system may internally cue a state that isequivalent to the state the system would have been in if the user hadprovided input (such as an utterance) containing all the components ofthe intent except for the needed information. In other words and in someexamples, the system may assume that the user has already provided moreinput, with that input missing only one or more specific slot(s)corresponding to the needed information. In this manner, the intenthandler 50 may continually utilize whatever user input is provided. Insome examples, this allows the system to reuse components, such asintent templates. Accordingly and in these examples, by causing theintent handler 50 to assume that user intents (versus system goals) aredriving its operation, the system may internally reuse correspondinglogic and may understand such user intents with greater depth andrichness.

In some examples, the system may have a goal of acquiring informationfrom a user to proceed with deriving a user intent. In a first example,a user may speak two utterances: “Book me a flight to Californiatomorrow; The flight needs to be to San Francisco.” In the firstutterance, the user indicates an intent to book a flight, and in thesecond utterance the user narrows the intent to a flight to SanFrancisco. In both utterances, a user intent is specified.

In another example, the user speaks a first utterance “Book me a flighttomorrow.” The system may respond with a query “Where do you want to flyto?” The user may then respond, “To San Francisco.” Upon generating thesystem query, the mapper 52 may map the intent handler's goal (acquiringinformation of the user's destination) to a user intent. For example,the mapper 52 may presume that the user is about to provide thisinformation as if it were the user's intent.

In some examples, by configuring the mapper 52 to presume that a userintent is driving its operation, the system may minimize the code toperform these operations and reuse corresponding logic. In this manner,the system may understand such user intents with greater depth andrichness. Accordingly, in these examples the system may utilize code forthe intent handler 50 and mapper 52 that comprises a user-intent onlysystem, as opposed to utilizing multiple specialized pieces of code tomanage all ambiguities and otherwise handle multiple corresponding tasksand discrete situations.

Additional details regarding components and computing aspects that maybe used to implement intent handler 50 are described in more detailbelow with respect to FIG. 23.

With reference now to FIGS. 6A and 6B, a flow chart of a method 300 foraddressing missing and/or unresolved information in an intent templateaccording to examples of the present disclosure is provided. Thefollowing description of method 300 is provided with reference to thesoftware and hardware components described herein. It will beappreciated that method 300 also may be performed in other contextsusing other suitable hardware and software components.

Additionally, while the blocks of method 300 are described andillustrated in a particular sequence, in different examples the order ofexecution may vary. In some examples one or more of the blocks may notbe performed. In some examples, context information 110 from the entitytracker 100 may be utilized to determine an order of execution and/orwhich block to execute next.

With reference to FIG. 6A, at 304 the method 300 may include performinganaphoric resolution on slots that contain an anaphor or a cataphor. Forexample, in the phrase “When he is near the oven alert me”, the word“he” is an anaphoric expression that refers to a person who wasreferenced earlier in the conversation. Additionally and as described inmore detail below, by understanding and resolving the intent of the uservia intent template(s) received from the parser 40, the intent handler50 may augment this anaphoric resolution process with one or more othertechniques, such as grounding and repair techniques described in moredetail below, to more accurately determine the person who corresponds toan anaphoric expression.

At 308 the method 300 may include performing deictic resolution on slotsthat contain words that cannot be fully understood without additionalcontextual information. Examples of deictic expressions include wordshaving a fixed semantic meaning and a denotational meaning that variesdepending on time and/or place. For example, in the phrase “When he isnear the oven alert me”, the word “near” is a deictic expression whosemeaning depends on contextual information. Additionally and as withanaphoric expressions, the intent handler 50 may augment its deicticresolution process with one or more other techniques, such as groundingand repair techniques, to clarify the intended meaning of the deicticexpression.

In some examples, deictic resolution may be performed using data fromone or more sensors, such as captured image data, audio data, positioninformation, etc. For example, when a user points at an oven, image datashowing the user's finger pointing at the oven may utilized by theentity tracker 100 to identify the oven and to determine that the useris pointing at the oven. While pointing the user may speak the utterance“Let me know when this gets hot.” Using this information, the intenthandler 50 may resolve the word “this” into “oven”, and moreparticularly into the oven at which the user is pointing.

In another example, a user may speak “If my child comes in here let meknow.” The system may use location data of the user to resolve the word“here” into the current location of the user. In another example, twopeople walk into the room, and one person asks the system: “Do we haveany messages?” Using sensory information, such as image data and/oraudio data to identify both people, the system may perform deicticresolution to resolve “we” to the identities of the two people in theroom.

At 312 the method 300 may include performing lookups for slotscontaining words that correspond to other data available to the intenthandler 50. Examples of other data that may be available to the intenthandler 50 include contact information, social graph data, calendarinformation, email data, photo metadata, and the like. Informationaccessed in performing such lookups may be populated in a slot toreplace the word(s) presently occupying the slot. For example, in thephrase “Tomorrow remind me to drop the car at autodealer1”, the word“autodealer1” may correspond to the auto repair shop where the userregularly has her car repaired. “Autodealer1” may be represented by acontact entry in the user's contact database. Accordingly, the intenthandler 50 may locate such contact entry and may utilize the“Autodealer1” contact data for the word “autodealer1” in the intenttemplate.

At this point, the method 300 may return to 304 and/or 308 to performanaphoric resolution and/or deictic resolution, as needed, oninformation populated in a slot. Additionally, the intent handler 50 mayaugment its lookup process with one or more other techniques, such asgrounding and repair techniques, to clarify the intended person orentity that corresponds to the information currently present in theslot.

At 314 the method 300 may include asking the user to disambiguate and/orresolve one or more slots containing ambiguous information. For example,where a user asks the system to “Call Patrick”, and the user's contactsdatabase includes a Patrick Doe and a Patrick Smith, the system may askthe user, “Which Patrick would you like to call, Patrick Smith orPatrick Doe?”

At 316 the method 300 may include identifying slots with missinginformation and populating these slots with collected information.Various techniques may be utilized to generate and/or retrieve suchinformation. For example and as described in more detail below, slotswith missing information may be treated differently depending uponwhether the information is determined to be explicitly missing orimplicitly missing.

For example, at 320 the method 300 may include determining that themissing information is explicitly missing information. In one example,by analyzing a text phrase the intent handler 50 may determine that theuser's utterance suggests that information for a particular slot shouldbe provided in the utterance. Where such information is missing, theinformation may be determined to be explicitly missing information. Forexample, consider the phrase “When Gary comes into the room with theothers introduce.” The intent handler 50 may determine that this phrasecomprises a content slot corresponding to the subject of the verb“introduce”, and that this content slot is missing information. In thisexample, the context of the phrase comprises the words that precede“introduce”, these words' order and meaning, the factor that the phraseends with the word “introduce” without naming the subject of theintroduction, and the factor that the phrase constitutes a grammaticallyincomplete sentence.

The intent handler 50 may determine that this context does not resolvethe ambiguity associated with this missing information. For example,while the user may be intending to introduce Gary to the others, otherintentions are also possible (such as introducing one of the others toGary). Accordingly, the intent handler 50 determines that the ambiguityassociated with this missing information cannot be presently resolved.Given this ambiguity and as described in more detail below, the intenthandler 50 may use one or more other techniques (such as querying theuser, “Whom do you want to introduce?”) to collect the missinginformation. In some examples as described in more detail below, theintent handler 50 may wait for the receipt of additional user inputbefore querying the user. In some examples, additional information fromthe entity tracker 100 may be utilized to resolve the ambiguity andcollect the missing information.

In some examples, where information for a trigger slot or an action slotof a commitment is missing, and based at least in part on contextinformation 110 generated by the entity tracker 100, the intent handler50 may proactively propose an action to the user. In one example, a usermay speak the utterance “Alice.” The intent handler 50 may receive anintent template with an empty action slot and a trigger slot partiallycompleted with the name “Alice.” The context information 110 may includean identity prediction with 85% confidence that “Alice” corresponds tothe “Alice Jones” in the user's contact database. The contextinformation 110 also may include a location prediction with 95%confidence that Alice Jones is located in the basement laundry room ofthe user's house. Based at least in part on this context information110, the intent handler 50 may proactively ask if the user would like tocommunicate with Alice Jones, such as via an in-home intercom system.

At 324 the method 300 may include determining that the missinginformation is implicitly missing information. In one example, theintent handler 50 may determine that a user did not intend to provideinformation that is missing from a particular slot. Such missinginformation may be determined to be implicitly missing information. Forexample, consider the phrase “When Gary walks into the kitchen sayHello.” The intent handler 50 may determine that the command “say Hello”corresponds to the user saying Hello to another person. Accordingly, theintent template corresponding to this phrase may comprise a content slotthat follows the words “say Hello” and which normally contains the nameor other identifier of the person the user intends to say Hello to(e.g., “Say Hello to Suzanne”).

In this example, because the phrase ended with the word “Hello”, suchcontent slot is missing information that identifies the person intended.The context of this phrase comprises the words that precede “Hello”,these words' order and meaning, and the factor that the phraseconstitutes a grammatically complete sentence. Given this context, theintent handler 50 infers that the user intends for the command “sayHello” to apply to Gary. In other words, the context associated withthis phrase indicates that the content slot following the words “sayHello” should be filled with “Gary.” In this manner, the intent handler50 may resolve this particular ambiguity associated with the missinginformation without querying the user for additional input. Afterpopulating a slot with missing information as described above, themethod 300 may return to 304 and 308 to perform anaphoric resolutionand/or deictic resolution, as needed, on the information populated inthe slot.

In some examples and as noted above, the intent handler 50 may query theuser for information that is missing from a slot. For example, theintent handler 50 may broadcast a spoken word query to the user via aspeaker of a mobile phone. In some examples, however, informationmissing from a slot may be the result of an intended or unintended pauseby the user that interrupts the user before the user completes herutterance. Accordingly and at 328, in some examples the method 300 mayinclude identifying a pause within an utterance from a user along withan intent of the user to continue speaking and add to the utterance.

For example, a user may pause mid-utterance to think about what sheshould say next. In other examples, a user may be interruptedmid-utterance by an external event, such as another person speaking,distracting activity from the user's environment such as a loud noise orbright light, or a variety of other external activities.

In one example and with reference to the description above foridentifying explicitly missing information, the phrase “When Gary comesinto the room with the others introduce” may be determined to comprise acontent slot that corresponds to the subject of the verb “introduce” andis missing information. Based on the empty content slot, other aspectsof the phrase, and/or the context in which it is spoken, the intenthandler 50 may identify a pause at the end of this phrase along with apredicted intent of the user to continue speaking and to add a subjectto the verb “introduce.”

At 332 and in response to identifying the pause, the method 300 mayinclude waiting for additional input from the user before asking theuser for more information. In some examples, the intent handler 50 maywait for a predetermined period of time, such as 1 second, 2 seconds, orother length of time that does not create a negative user experience forthe user. In this manner, the system may avoid interrupting the usermid-utterance where the user intends to begin speaking again and to addto the utterance.

In some examples, an engagement timer may be started whenever a userstarts speaking. The engagement timer may run for a predetermined periodof time during which the user may be designated as “engaged.” Thepredetermined period of time may be 1 second, 2 seconds, or otherduration. If the system needs to ask for input or otherwise audiblyconverse with the user before the predetermined period of time expires,the system may use interruption language constructs that may provide fora more gentle interruption of the user's current potential engagement.Examples of interruption language constructs include “by the way” and“additionally”. In some examples, such language constructs may be usedeven where the user has stopped speaking and/or the current conversationhas “timed out,” and the system is not presently listening to the user.

At 336 the method 300 may include querying the user for informationmissing from a slot. In some examples, the intent handler 50 may ask theuser for information missing from one or more slots of an intenttemplate. For example, regarding the phrase “When Gary comes into theroom with the others introduce” and its explicitly missing informationin the content slot following the word “introduce,” the intent handler50 may broadcast a spoken word query to the user asking “Whom do youwant to introduce?” In other examples, the intent handler 50 may querythe user via other interfaces, such as by displaying a query on adisplay device.

When the intent handler 50 receives a response to its query from theuser (via the voice listener 30 and parser 40), the intent handler maypopulate the slot with the response. At this point, the method 300 mayreturn to 304 and the steps following to analyze this newly-addedinformation for any ambiguities as described above.

With reference now to FIG. 6B, at 340 the method 300 may includeresolving an ambiguity by analyzing information from a priorconversation turn. In different examples, the method may analyze bothutterances as a single or combined utterance, and/or may use one or moreelements from a prior utterance to generate one or more slots in anintent template for a current utterance.

In some examples, the intent handler 50 may analyze content from aprevious intent template and/or one or more slots of the template. Insome examples, the intent handle 50 may determine that a currentutterance is additive to a previous utterance. For example, consider thephrase “When Justin is near the oven, alert Erich.” Justin may be atoddler, Erich the toddler's father, and the user speaking the phrasemay be Justin's mother. The intent handler 50 may receive a first intenttemplate for this phrase. A first resolver may resolve the template andestablish a commitment that broadcasts a warning to Erich via Erich'smobile phone when Justin is within 1 meter of the oven in Erich'skitchen.

After speaking this first phrase, Justin's mother may pause for a briefperiod of time, such as 3 or 4 seconds. After this pause, she may speaka second phrase “and me” which is received by the parser 40. As thisphrase contains no action component, the parser 40 may generate a secondintent template that has an unknown or unresolved intent. In thisexample, and because the intent associated with this second phrase ispresently unknown, the intent handler 50 may select a second, differentresolver to address this second intent template.

Based at least in part on this second phrase beginning with theconjunction “and” followed by the pronoun “me”, the second resolver maydetermine that Justin's mother intends to refer to a prior utterance.The second resolver may utilize an anaphoric resolution technique toassociate the word “me” to Justin's mother. By using this data andanalyzing the previously-established commitment, the second resolver maydetermine that the intent associated with the second phrase “and me” isrelated to the intent associated with the prior phrase “When Justin isnear the oven, alert Erich.” Accordingly, the second resolver may modifythe previously-established commitment to broadcast a warning to bothErich and Justin's mother when Justin is within 1 meter of the oven inthe kitchen.

As another example, consider again the phrase “When Justin is near theoven, alert Erich.” After speaking this first phrase, Justin's mothermay pause for a few seconds and then speak a second phrase “and also ifhe's close to the pool.” As this phrase contains a trigger (“if he'sclose to the pool”) and no action component, the parser 40 may generatea second intent template that has an unknown or unresolved intent. Also,in this example the anaphoric expression “he's” could refer to either ofthe two names in the preceding phrase (Justin or Erich).

A resolver may determine that it is most probable that the reference to“he” in the trigger of the second phrase is intended to refer to a maleperson mentioned in another, prior trigger. Based at least in part onthis second phrase beginning with the conjunction “and” followed by thewords “also” and “if”, the second resolver may determine that Justin'smother intends to refer to a prior utterance and to modify a trigger oradd another trigger to an action of the previously-establishedcommitment. By using this data and analyzing the previously-establishedcommitment, the second resolver may determine that the intent associatedwith the second phrase “And also if he's close to the pool” is relatedto the intent associated with the prior phrase “When Justin is near theoven, alert Erich.” Accordingly, the second resolver may modify thepreviously-established commitment to broadcast a warning to Erich whenJustin is either within 1 meter of the oven in the kitchen or within 3meters of the pool.

In some examples, the intent handle 50 may determine that a currentutterance is intended to amend one or more previous utterances. Forexample, consider the phrase “Please remind me to call Jeff at sixo'clock.” After speaking this first phrase, the user may pause for abrief moment and then speak a second phrase “I mean Mike.” As thisphrase contains an ambiguous phrase without a clear trigger or actioncomponent, the parser 40 may generate another intent template that hasan unresolved intent.

By analyzing the immediately preceding commitment associated with theprior utterance “Please remind me to call Jeff at six o'clock,” aresolver may determine that the intent associated with the second phrase“I mean Mike” is most likely related to the intent associated with theprior phrase “Please remind me to call Jeff at six o'clock.”Accordingly, this resolver may modify the previously-establishedcommitment to replace the reference to “Jeff” in the action component ofthis phrase with “Mike.”

In another example, consider the phrase “Please remind me to call Jeffand Mike at six o'clock.” After speaking this first phrase, the user maypause for a brief moment and then speak a second phrase “not Mike.” Asthis phrase contains an ambiguous phrase without a clear trigger oraction component, the parser 40 may generate another intent templatethat has an unresolved intent.

By analyzing the immediately preceding commitment associated with theutterance “Please remind me to call Jeff and Mike at six o'clock,” aresolver may determine that the intent associated with the second phrase“not Mike” is most likely related to the intent associated with theprior phrase “Please remind me to call Jeff and Mike at six o'clock.”Accordingly, this resolver may modify the previously-establishedcommitment to remove the reference to “and Mike” from the actioncomponent of this phrase.

In some examples and as described in more detail below, where two ormore people are having a conversation, the system may follow theconversation and determine when the active participant (i.e., the personcurrently speaking) changes in the conversation. In these examples, whenthe system determines that the current speaker has changed, the systemmay determine whether the information contained in the new speaker'sspeech is a continuation of the existing conversation topic/session, orwhether a new topic/session has been introduced. Where the new speaker'sinformation is a continuation of the existing conversationtopic/session, this determination may be used by the intent handler 50to resolve ambiguities, complete missing information and/or otherwiseclarify the intent of each speaker. For example, such conversation andtopic/session tracking may enable the system to assist a team that isworking and speaking collaboratively to complete a task. In someexamples, the system may track multiple conversations that are occurringsimultaneously or otherwise overlapping, and may interact withparticipants in each conversation as appropriate for each conversation.

In some examples, the intent handler 50 may determine that an intentassociated with a newly received phrase is not related to the intent ofan immediately preceding commitment. For example, an intent templatecorresponding to the utterance “Call Justin” may be received andprocessed by a first resolver into a first commitment. The firstresolver may determine that the content slot (“Justin”) of the action“Call Justin” is ambiguous because the user has both a Justin Smith anda Justin Doe in the user's contacts database. Accordingly, the firstresolver may respond with a query to the user of “Which Justin—JustinDoe or Justin Smith?” In this example, the user responds with anunrelated response, “Please record TV Show A tonight.”

The first resolver may analyze this response and its corresponding newintent template by referring to the immediately preceding intenttemplate and its missing content slot. Because the user's response iscompletely unrelated to the query just presented to the user, the firstresolver determines that the new intent template represents a new intentof the user, and thus the new intent template should not be fused withthe preceding intent template. Accordingly, the first resolver isreplaced by a second resolver that proceeds to analyze the new intenttemplate and establish a new conversation.

At 344 the method 300 may include resolving an alias that refers to aknown person or entity by a different name or representation. In oneexample, a user may refer to “Mimi” in an utterance. The user's contactsdatabase may not contain a contact with the name “Mimi.” However, inprior conversations tracked by the intent handler 50, the user's sistermay have referred to herself as “Mimi” when speaking with her grandson.A data store accessible to the intent handler 50 may have created anassociation between the user's sister and the alias “Mimi.” By searchingthe data store for instances of “Mimi” and finding the associationbetween the user's sister and the alias “Mimi”, the intent handler 50may resolve the name “Mimi” in the user's utterance to the user'ssister.

At 348 the method 300 may include training the intent handler 50 toassociate an alias with a known person or other entity via direct usertraining input. For example, the user may speak a command, “When I sayMimi I'm referring to my sister Suzanne.” The intent handler 50 maycreate a link between “Mimi” and the user's sister Suzanne, such as bymodifying a contacts database file containing information identifyingSuzanne.

In a similar manner, at 352 the method 300 may include training theintent handler 50 in a real-time or batch-mode manner to correlate anunrecognized surface form with a newly resolved surface form. Forexample, the intent handler 50 may be unable to recognize a particularsurface form it receives. The intent handler 50 may clarify this surfaceform via one or more grounding and repairing techniques. In this mannerand going forward, the unrecognized surface form subsequently may becorrelated with the clarified surface form, whereby the intent handler50 now may recognize the previously-unrecognized surface form.

In another example, a user may be traveling across New York City in acar for hire. The user may speak a first request to his smartphone, witha middle portion of the phrase unintelligible: “When I get to[unintelligible] call her mobile phone.” By analyzing this phrase alongwith context information, such as motion data indicating the user istraveling in a car, the intent handler 50 may infer that theunintelligible portion of the phrase corresponds to a location slot.

The intent handler 50 may query the user, “Where do you want to dothis?” The user may reply with a second response, “Madison.” The parser40 may receive the text “Madison” from the voice listener 30, and maygenerate a list of the statistically most probable meanings for thisword that correspond to the user's actual intent. In this example, theuser may have a close friend named Madison, and may have used her namein many spoken requests to the intelligent assistant system 20.Accordingly, the parser 40 may determine that the user's close friend“Madison” is the most probable intention underlying the user'sutterance.

However, based its analysis of the user's first request and othercontext information, such as the motion data, the intent handler 50determines that the expected user response to the query “Where do youwant to do this?” most likely will be location information. The intenthandler also may analyze mapping data that indicates the user willarrive at a Madison Avenue address in five minutes. Accordingly andbased at least in part on this context information, the intent handler50 may not select the user's close friend “Madison”, despite theparser's prediction that this is the statistically most probable meaningfor this word. Instead, the intent handler may use this contextinformation to resolve this ambiguity by selecting Madison Avenue as theintention of the user.

In some examples where the intent handler is unable to resolve an intentfrom an utterance, the system may still offer to take one or moreactions. For example, if a user makes the declarative statement “Silverlooks nice”, the system may not understand the user's intent underlyingthis utterance. Instead of ignoring the user because the system doesn'tunderstand what it should do with the utterance, the system may offer todisplay photos of silver jewelry, play music, or take some other action.

It will be appreciated that method 300 is provided by way of example andis not meant to be limiting. Therefore, it is to be understood thatmethod 300 may include additional and/or alternative steps relative tothose illustrated in FIGS. 6A and 6B. Further, it is to be understoodthat method 300 may be performed in any suitable order. Further still,it is to be understood that one or more steps may be omitted from method300 without departing from the scope of this disclosure.

As described above, when the intent handler 50 has sufficientlyclarified and resolved the user's intent, a corresponding commitment maybe generated and passed to the commitment engine 60 for execution. Asdescribed in more detail below, the commitment engine 60 may utilize oneor more cost functions to determine one or more costs associated withexecuting or not executing a commitment and, in some examples, withoutputting or not outputting a message to the user.

Where the commitment engine 60 receives multiple commitments, thecommitment engine may prioritize the commitments for presentation to auser. In one use case example, the commitment engine 60 may be storingseven commitments for user Eddie, with each commitment set to convey adifferent message to Eddie. Each message also may be staged to beaudibly broadcast to Eddie when he arrives home from work today. Thecommitments and corresponding messages may include task reminders totake out the garbage, fix the leaky faucet and put the roast in theoven, and informational reminders that property taxes are due tomorrowand that he needs to buy eggs. If each of these messages is broadcastedto Eddie when he walks in the door, he may be less likely to effectivelymanage and/or prioritize the corresponding tasks.

Additionally, in some examples Eddie's current context may make it moredifficult for him to comprehend and effectively manage these messages.For example, if Eddie is talking on his phone when he walks in the door,and seven messages are audibly broadcast to him at that time, he mayhave difficulty hearing or even comprehending the messages.

Accordingly and in some examples, factors related to the receptivity ofthe user to receiving input, the importance of a commitment to aparticular user, and/or the user's current context may be determined.Machine learning techniques may be applied to such factors and otherdata to learn from such information and make related predictions in thefuture. As described in more detail below, one or more cost functionsmay be used to determine costs associated with executing or notexecuting a commitment. Using these techniques, the commitment engine 60may intelligently manage the execution of commitments and correspondingmessages to align with a particular user's preferences and currentcontext.

In some examples, and in response to changing contexts and/or new datainputs, the commitment engine 60 may modify priorities, timings, andother aspects of commitments, messages and their execution. For exampleand as described in more detail below, the commitment engine 60 mayreceive context information 110, such as entity identity, entityposition, and entity status information, from the entity tracker 100.Such context information 110 may be used by commitment engine 60 todetermine whether a particular message, notification, or commitmentshould be presented to a user or otherwise executed.

In some examples, one or more previously defined components of acommitment may be updated based on new input received by the commitmentengine 60. For example, the intent handler 50 may generate and pass acommitment including a trigger component that refers to aseparately-defined term. In one example, a user may speak the utterance:“Please notify my kids to come home 60 minutes before curfew.” The term“curfew” may be associated with the user's profile that is stored in adata store, and may currently have a value of 11:00 pm. By accessing theuser's profile stored in a data store, the intent handler 50 may resolvethe term “curfew” to 11:00 pm, and may pass to the commitment engine 60a corresponding commitment to send a text message at 10:00 pm (60minutes before 11:00 pm) to the user's children with instructions tocome home.

Subsequently to this utterance, the user may update her kids' curfewtime to one hour later, such as by speaking: “Update the kids' curfew toMidnight.” The commitment engine 60 may identify this update itsmodification to the value of “curfew,” and may determine that it affectsthe previously-received commitment. Accordingly, the commitment enginemay correspondingly modify the trigger of the previously-receivedcommitment by updating the value of “curfew” to Midnight, which resultsin the commitment sending the text message at 11:00 pm instead of 10:00pm. The commitment engine 60 also may modify the value of “curfew” inthe user's profile stored in the data store.

Additional details regarding components and computing aspects that maybe used to implement commitment engine 60 are described in more detailbelow with respect to FIG. 23.

FIG. 7 schematically illustrates an example entity tracker 100 that maycomprise a component of the intelligent assistant system 20. Entitytracker 100 may be used to determine an identity, position, and/orcurrent status of one or more entities within range of one or moresensors. Entity tracker 100 may output such information to one or moreother modules of intelligent assistant system 20, such as the commitmentengine 60, voice listener 30, etc.

The word “entity” as used in the context of the entity tracker 100 mayrefer to people, animals, or other living things, as well as non-livingobjects. For example, the entity tracker may be configured to identifyfurniture, appliances, structures, landscape features, vehicles, and/orany other physical object, and determine the position/location andcurrent status of such physical objects. In some cases, the entitytracker 100 may be configured to only identify people and not otherliving or non-living things. In such cases, the word “entity” may besynonymous with the word “person.”

Entity tracker 100 receives sensor data from one or more sensors 102,such as sensor A 102A, sensor B 102B, and sensor C 102C, though it willbe understood that an entity tracker may be used with any number andvariety of suitable sensors. As examples, sensors usable with an entitytracker may include cameras (e.g., visible light cameras, UV cameras, IRcameras, depth cameras, thermal cameras), microphones, pressure sensors,thermometers, motion detectors, proximity sensors, accelerometers,global positioning satellite (GPS) receivers, magnetometers, radarsystems, lidar systems, environmental monitoring devices (e.g., smokedetectors, carbon monoxide detectors), barometers, health monitoringdevices (e.g., electrocardiographs, sphygmomanometers,electroencephalograms), automotive sensors (e.g., speedometers,odometers, tachometers, fuel sensors), and/or any other sensors ordevices that collect and/or store information pertaining to theidentity, position, and/or current status of one or more people or otherentities. In some examples, the entity tracker 100 may occupy a commondevice housing with one or more of the plurality of sensors 102, and/orthe entity tracker and its associated sensors may be distributed acrossmultiple devices configured to communicate via one or more networkcommunications interfaces (e.g., Wi-Fi adapters, Bluetooth interfaces).

As shown in the example of FIG. 7, entity tracker 100 may include anentity identifier 104, a person identifier 105, a position (location)identifier 106, and a status identifier 108. In some examples, theperson identifier 105 may be a specialized component of the entityidentifier 100 that is particularly optimized for recognizing people, asopposed to other creatures and non-living things. In other cases, theperson identifier 105 may operate separately from the entity identifier104, or the entity tracker 100 may not include a dedicated personidentifier.

Depending on the specific implementation, any or all of the functionsassociated with the entity identifier, person identifier, positionidentifier, and status identifier may be performed by the individualsensors 102A-102C. Though the present description generally describesthe entity tracker 100 as receiving data from sensors, this does notrequire that the entity identifier 104, as well as other modules of theentity tracker, must be implemented on a single computing device that isseparate and distinct from the plurality of sensors associated with theentity tracker. Rather, functions of the entity tracker 100 may bedistributed amongst the plurality of sensors. For example, rather thansending raw sensor data to the entity tracker, individual sensors may beconfigured to attempt to identify entities that they detect, and reportthis identification to the entity tracker 100, and/or other modules ofintelligent assistant system 20. In some cases, this identification mayinclude a confidence value.

Each of the entity identifier 104, person identifier 105, positionidentifier 106, and status identifier 108 is configured to interpret andevaluate sensor data received from the plurality of sensors 102, and tooutput context information 110 based on the sensor data. Contextinformation 110 may include the entity tracker's guesses/predictions asto an identity, position, and/or status of one or more detected entitiesbased on received sensor data. As will be described in more detailbelow, each of the entity identifier 104, person identifier 105,position identifier 106, and status identifier 108 may output theirpredictions/identifications along with a confidence value.

The entity identifier 104 may output an entity identity 112 of adetected entity, and such entity identity may have any suitable degreeof specificity. In other words, based on received sensor data, theentity tracker 100 may predict the identity of a given entity, andoutput such information as entity identity 112. For example, the entityidentifier 104 may report that a particular entity is a piece offurniture, a dog, a human male, etc. Additionally, or alternatively, theentity identifier 104 may report that a particular entity is an ovenwith a particular model number; a pet dog with a specific name andbreed; an owner or user of intelligent assistant system 20, with theowner/user having a particular name and profile; etc. In some examples,the degree of specificity with which the entity identifier 104identifies/classifies detected entities may depend on one or more ofuser preferences and sensor limitations.

When applied to people, the entity tracker 100 may in some cases collectinformation about individuals whom it is unable to identify by name. Forexample, the entity identifier 104 may record images of a person's face,and associate these images with recorded audio of the person's voice.Should the person subsequently speak to or otherwise address theintelligent assistant system 20, the entity tracker 100 will then haveat least some information regarding with whom the intelligent assistantsystem is interacting. In some examples, the intelligent assistantsystem 20 could also prompt the person to state their name, so as tomore easily identify the person in the future.

In some examples, the intelligent assistant system 20 may utilize aperson's identity to customize a user interface for the person. In oneexample, a user may be identified who has limited visual capabilities.In this example and based on this identification, a display of theintelligent assistant system 20 (or other device with which the user isinteracting) may be modified to display larger text, or to provide avoice-only interface.

The position identifier 106 may be configured to output an entityposition (i.e., location) 114 of a detected entity. In other words, theposition identifier 106 may predict the current position of a givenentity based on collected sensor data, and output such information asentity position 114. As with the entity identity 112, the entityposition 114 may have any suitable level of detail, and this level ofdetail may vary with user preferences and/or sensor limitations. Forexample, the position identifier 106 may report that a detected entityhas a two-dimensional position defined on a plane such as a floor orwall. Additionally, or alternatively, the reported entity position 114may comprise a three-dimensional position of a detected entity within areal world, three-dimensional environment. In some examples an entityposition 114 may comprise a GPS position, a location within a mappingsystem, etc.

The reported entity position 114 for a detected entity may correspond tothe entity's geometric center, a particular part of the entity that isclassified as being important (e.g., the head of a human), a series ofboundaries defining the borders of the entity in three-dimensionalspace, etc. The position identifier 106 may further calculate one ormore additional parameters describing the position and/or orientation ofa detected entity, such as a pitch, roll, and/or yaw parameter. In otherwords, the reported position of a detected entity may have any number ofdegrees-of-freedom, and may include any number of coordinates definingthe position of the entity in an environment. In some examples, anentity position 114 of a detected entity may be reported even if theentity tracker 100 is unable to identify the entity, and/or determinethe current status of the entity.

Status identifier 108 may be configured to output an entity status 116of a detected entity. In other words, the entity tracker 100 may beconfigured to predict the current status of a given entity based onreceived sensor data, and output such information as entity status 116.“Entity status” can refer to virtually any measurable or classifiableproperty, activity, or behavior of a given entity. For example, whenapplied to a person, the entity status of the person can indicate aposture of the person (e.g., standing, sitting, laying down), a speed atwhich the person is walking/running, a current activity of the person(e.g., sleeping, watching TV, working, playing a game, swimming, talkingon the phone), a current mood of the person (e.g., by evaluating theperson's facial expression or tone of voice), biological/physiologicalparameters of the person (e.g., the person's heart rate, respirationrate, oxygen saturation, body temperature, neurological activity),whether the person has any current or upcoming calendarevents/appointments, etc. “Entity status” can refer toadditional/alternative properties or behaviors when applied to othercreatures or non-living objects, such as a current temperature of anoven or kitchen sink, whether a device (e.g., television, lamp,microwave) is powered on, whether a door is open, etc.

In some examples, the status identifier 108 may use sensor data tocalculate a variety of different biological/physiological parameters ofa human. This may be done in a variety of suitable ways. For example,the entity tracker 100 may be configured to interface with an opticalheart rate sensor, a pulse oximeter, a sphygmomanometer,electrocardiograph, etc. Additionally or alternatively, the statusidentifier 108 may be configured to interpret data from one or morecameras and/or other sensors in an environment, and process the data inorder to calculate a human's heart rate, respiration rate, oxygensaturation, etc. For example, the status identifier 108 may beconfigured to utilize Eulerian magnification and/or similar techniquesto amplify miniscule movements or changes captured by the cameras,thereby allowing the status identifier to visualize the flow of bloodthrough a human's circulatory system and calculate associatedphysiological parameters. Such information can be used, for example, todetermine when the person is asleep, working out, in distress,experiencing health problems, etc.

Upon determining one or more of the entity identity 112, entity position114, and entity status 116, such information may be sent as contextinformation 110 to any of a variety of external modules or devices,where it may be used in a variety of ways. For example, contextinformation 110 may be used by commitment engine 60 to managecommitments and associated messages and notifications. In some examplesand as described in more detail below, context information 110 may beused by commitment engine 60 to determine whether a particular message,notification, or commitment should be executed and/or presented to auser. Similarly, context information 110 may be utilized by voicelistener 30 when interpreting human speech or activating functions inresponse to a keyword trigger.

As noted above, in some examples the entity tracker 100 may beimplemented in a single computing device. In other examples, one or morefunctions of the entity tracker 100 may be distributed across multiplecomputing devices working cooperatively. For example, one or more of theentity identifier 104, person identifier 105, position identifier 106,and status identifier 108 may be implemented on different computingdevices, while still collectively comprising an entity trackerconfigured to perform the functions described herein. As indicatedabove, any or all of the functions of the entity tracker may beperformed by individual sensors 102. Further, in some examples entitytracker 100 may omit one or more of the entity identifier 104, personidentifier 105, position identifier 106, and status identifier 108,and/or include one or more additional components not described herein,while still providing context information 110. Additional detailsregarding components and computing aspects that may be used to implemententity tracker 100 are described in more detail below with respect toFIG. 23.

Each of entity identity 112, entity position 114, and entity status 116may take any suitable form. For example, each of the entity identity112, position 114, and status 116 may take the form of a discrete datapacket including a series of values and/or labels describing theinformation gathered by the entity tracker. Each of the entity identity112, position 114, and status 116 may additionally include a confidencevalue defining a statistical likelihood that the information isaccurate. For example, if the entity identifier 104 receives sensor datathat strongly indicates that a particular entity is a human male named“John Smith,” then entity identity 112 may include this informationalong with a corresponding relatively high confidence value, such as 90%confidence. If the sensor data is more ambiguous, then the confidencevalue included in entity identity 112 correspondingly may be relativelylower, such as 62%. In some examples, separate predictions may beassigned separate confidence values. For example, the entity identity112 may indicate with 95% confidence that a particular entity is a humanmale, and indicate with a 70% confidence that the entity is John Smith.As described in more detail below, such confidence values (orprobabilities) may be utilized by a cost function in generating costcalculations for providing messages or other notifications to a userand/or performing action(s).

In some implementations, the entity tracker 100 may be configured tocombine or fuse data from multiple sensors in order to output moreaccurate predictions. As an example, a camera may locate a person in aparticular room. Based on the camera data, the entity tracker 100 mayidentify the person with a confidence value of 70%. However, the entitytracker 100 may additionally receive recorded speech from a microphone.Based on the recorded speech alone, the entity tracker 100 may identifythe person with a 60% confidence value. By combining the data from thecamera with the data from the microphone, the entity tracker 100 mayidentify the person with a higher confidence value than would bepossible using the data from either sensor alone. For example, theentity tracker may determine that the recorded speech received from themicrophone corresponds to lip movements of the person visible to thecamera when the speech was received, and thereby conclude withrelatively high confidence, such as 92%, that the person visible to thecamera is the person speaking. In this manner the entity tracker 100 maycombine the confidence values of two or more predictions to identify aperson with a combined, higher confidence value.

In some examples, data received from various sensors may be weighteddifferently depending upon a reliability of the sensor data. This can beespecially relevant in situations where multiple sensors are outputtingseemingly inconsistent data. In some examples, the reliability of asensor's data may be based at least in part on the type of datagenerated by the sensor. For example, in some implementations areliability of video data may be weighted higher than a reliability ofaudio data, as the presence of an entity on camera may be a betterindicator of its identity, position, and/or status than recorded soundsthat are presumed to originate from the entity. It will be appreciatedthat a reliability of sensor data is a different factor than aconfidence value associated with a predicted accuracy of an instance ofdata. For example, several instances of video data may have differentconfidence values based on different contextual factors present at eachinstance. Each of these instances of video data, however, may beassociated with a single reliability value for video data in general.

In one example, data from a camera may suggest that a particular personis in a kitchen with a 70% confidence value, such as via facerecognition analysis. Data from a microphone may suggest with a 75%confidence value that the same person is in a nearby hallway, such asvia voice recognition analysis. Even though the instance of microphonedata carries a higher confidence value, the entity tracker 100 mayoutput a prediction that the person is in the kitchen based on a higherreliability of the camera data as compared to a lower reliability of themicrophone data. In this manner and in some examples, differentreliability values for different sensor data may be used along withconfidence values to reconcile conflicting sensor data and determine anidentity, position, and/or status of an entity.

Additionally or alternatively, more weight may be given to sensors thathave higher precision, more processing power or otherwise greatercapabilities. For example, a professional-grade video camera may have asignificantly improved lens, image sensor, and digital image processingcapabilities as compared to a basic webcam found in a laptop.Accordingly, a higher weight/reliability value may be given to videodata received from the professional-grade camera as compared to thewebcam, as such data is likely to be more accurate.

With reference now to FIG. 8, in some examples, individual sensors usedwith the entity tracker 100 may output data with a different frequencythan other sensors used with the entity tracker. Similarly, sensors usedwith the entity tracker 100 may output data with a different frequencythan the frequency with which the entity tracker evaluates the data andoutputs context information. In the example of FIG. 8, entity tracker100 may receive and interpret sensor data over multiple time frames200A, 200B, and 200C. A single time frame may represent any suitablelength of time, such as 1/30th sec., 1/60th sec., etc.

In this example, during time frame 200A entity tracker 100 receives aset of sensor data 202 including sensor A data 204A, sensor B data 204B,and sensor C data 204C. Such sensor data is interpreted by entitytracker 100 and transformed into context information 206, which may beused to determine an identity, position, and/or status of one or moredetected entities as described above. During time frame 200B, entitytracker 100 receives sensor data 208, including sensor A data 210A andsensor B data 210B. Entity tracker 100 does not receive data from sensorC during time frame 200B, as sensor C outputs data at a differentfrequency than sensors A and B. Similarly, entity tracker 100 does notoutput context information during time frame 200B, as the entity trackeroutputs context information at a different frequency than sensors A andB.

During time frame 200C, entity tracker 100 receives sensor data 212,including sensor A data 214A, sensor B data 214B, sensor C data 214C,and sensor D data 214D. Entity tracker 100 also outputs contextinformation 216 during time frame 200C, which may be based on any or allof the sensor data received by the entity tracker since contextinformation was last output in time frame 200A. In other words, contextinformation 216 may be based at least in part on sensor data 208 as wellas sensor data 212. In some examples, context information 216 may bebased at least in part on sensor data 202 and sensor data 208, as wellas sensor data 212.

As shown in FIG. 8, after the entity tracker 100 receives data from aparticular sensor, multiple time frames may pass before the entitytracker receives more data from the same sensor. During these multipletime frames, entity tracker 100 may output context information.Similarly, the usefulness of data received from a particular sensor mayvary from time frame to time frame. For example, at a first time framethe entity tracker 100 may receive audio data of a particular personspeaking via a microphone, and accordingly identify an entity position114 of the person with a relatively high confidence value. In subsequenttime frames, the person may remain at the identified position, but alsomay have stopped speaking since the first time frame. In this case, theabsence of useful data from the microphone may not be a reliableindicator of the absence of the person. Similar issues can arise withother types of sensors. For example, a camera may lose track of a personif he covers his face, or is occluded by an obstacle, such as anotherperson or a moving object. In this case, though current camera data maynot suggest the presence of the person, prior instances of camera datamay suggest that the person is still located at the previouslyidentified position. In general, while sensor data may reliably indicatethe presence of an entity, such data may be less reliable in suggestingthe absence of an entity.

Accordingly, the entity tracker 100 may utilize one or more confidencedecay functions, which in different examples may be defined by theentity tracker and/or by the sensors themselves. A confidence decayfunction may be applied to sensor data to reduce the entity tracker'sconfidence in the data from a particular sensor as time passes sincethat sensor last positively detected an entity. As an example, after asensor detects an entity at a particular location, the entity tracker100 may report context information 110 indicating that the entity is atthe location with relatively high confidence. If after one or more timeframes the sensor no longer detects the entity at the location, andunless it subsequently gathers contradictory evidence, the entitytracker 100 still may report that the entity is at the location, thoughwith a somewhat lower confidence. As time continues to pass since thesensor last detected the entity at the location, it becomesprogressively less likely that the entity is still at the location.Accordingly, the entity tracker 100 may utilize the confidence decayfunction to progressively decrease the confidence value of its reportedcontext information 110, eventually reaching 0% confidence if noadditional sensors detect the entity.

In some cases, different confidence decay functions may be utilized withdifferent sensors and sensor types. A selection of a particular decayfunction may depend at least in part on particular properties of asensor. For example, confidence values associated with data from a videocamera may decay more rapidly than confidence values associated withdata from a microphone, as absence of an entity in a video frame is amore reliable indicator of the entity's absence than silence recorded bya microphone.

One example of sensor confidence decay is schematically illustrated inFIG. 9, which shows entity tracker 100 receiving sensor data duringthree different time frames 300A, 300B, and 300C. During time frame300A, entity tracker 100 receives camera data 302 in which an entity isvisible in the frame. Based on this data, the entity tracker 100 reportsthe entity position 304 with a 90% confidence value. In time frame 300B,entity tracker 100 receives camera data 306 in which the entity is nolonger visible in the frame. However, it is possible that the entity hasnot moved, and has merely become occluded, or otherwise undetectable tothe camera. Accordingly, entity tracker 100 reports the same entityposition 304, but with a lower confidence value of 80%.

Finally, in time frame 300C entity tracker 100 receives camera data 310indicating that the entity is still not visible in the frame. As timehas passed, it has grown less likely that the entity is still in thesame position. Accordingly, the entity tracker 100 reports the sameentity position 304 with a still lower confidence value of 60%.

In some examples, variable reliability of sensor data may be at leastpartially addressed by making use of data filtering techniques. In someexamples, a Kalman filter may be utilized to filter sensor data. AKalman filter is a mathematical function that may combine multipleuncertain measurements and output a prediction with more confidence thanwould be possible using any individual measurement. Each measurementinput to the Kalman filter is given a weight based on the measurement'sperceived reliability. Kalman filters operate in a two-step process,including a prediction step and an update step. During the predictionstep, the filter outputs a prediction based on recent weightedmeasurements. During the update step, the filter compares its predictionto an actual observed value or state, and dynamically adjusts theweighting applied to each measurement so as to output more accuratepredictions.

In some examples, entity tracker 100 may comprise a Kalman filter thatcombines data from a variety of sensors to compensate for lower sensorreliability, such as when sensor confidence values have decayed overtime since the last positive detection. In some examples, entity tracker100 may apply a Kalman filter to sensor data when one or more sensorconfidence values are below a predetermined threshold. In an examplescenario, image data from a camera may be analyzed using face detectiontechniques to reliably detect a person in a particular room. Inresponse, the entity tracker 100 may report with high confidence thatthe person is located in the room.

In subsequent time frames, the camera may no longer be able to captureand/or positively recognize the person's face in the room. For example,the person's face may become occluded, or the camera may transmit datawith a much lower frequency than the entity tracker 100 outputs contextinformation 110. If the entity tracker 100 relied exclusively on datafrom the camera, then the confidence value of its reported position ofthe person would gradually decrease until the next positive detection.However and in some examples, data from the camera may be supplementedwith data from other sensors. For example, during the subsequent timeframes a microphone may report that it hears the person's voice in theroom, or another sensor may report that it can detect the presence ofthe person's mobile device in the room. In such cases, this data may beassigned weights by the Kalman filter, and may be used to predict theperson's current location with more confidence than would be possible ifonly the camera data were used.

In some cases, detection of people and/or other entities in anenvironment can become more complicated when sensor data is contaminatedby background information. Such background information may compromisethe confidence with which the entity tracker 100 reports entity identity112, position 114, and/or status 116. For example, the intelligentassistant system 20 may need to determine the identity of a person whois speaking in order to appropriately respond to a query or command.Such a determination can be difficult when multiple people are speakingat the same time, a television is playing, loud machinery is operating,etc.

Accordingly, the entity tracker 100 may use a variety of audioprocessing techniques to more confidently identify a particular activeparticipant who is engaged in a conversation with other people and/orwith the intelligent assistant system 20. As an example, the entitytracker 100 may implement a voice activity detection (VAD) engine thatmay distinguish human voices from environmental noise, and identify thepresence or absence of human speech.

General-purpose VAD engines may be used for the purpose of classifying aparticular segment of audio as including either speech or non-speech,with a corresponding confidence value. An entity tracker 100 also mayutilize a speaker recognition engine to match a particular audio segmentwith a particular person. As more speech is received, the speakerrecognition engine may be progressively tailored to classify the audioas including speech from a particular conversation participant, or notincluding speech from the particular conversation participant. In thismanner, the entity tracker 100 may recognize speech from one or moreparticular persons/conversation participants.

Training of a speaker recognition engine may occur any time the entitytracker 100 has confidently identified a particular person and recordedaudio that can be confidently attributed to that person. For example,using camera data, the entity tracker 100 may identify a particularperson and determine that the person's lips are moving. The entitytracker 100 may simultaneously receive audio from a microphone that canbe safely assumed to include speech from the identified person.Accordingly, the received audio can be used to retrain the speakerrecognition engine to more specifically recognize the identifiedperson's voice.

In some cases, such retraining may occur only when the person has beenidentified with a high confidence value (e.g., via accurate facialrecognition or any other method), such as a confidence value exceeding apredetermined threshold, and when the entity tracker 100 has received anaudio recording of the person's voice having high volume/amplitude and ahigh signal-to-noise ratio (S/N). Using this technique, the entitytracker 100 may accumulate a variety of person-specific voice models,allowing the entity tracker to more consistently identify speech fromparticular people and ignore background noise.

With reference now to FIG. 10, an example of using a trained speechrecognition engine to recognize speech from a particular person isschematically illustrated. In this example, entity tracker 100 receivestwo speech fragments 400A and 400B. Speech fragment 400A includesrecorded speech of a person 1, and speech fragment 400B includesrecorded speech of a person 2. Entity tracker 100 includes a speechrecognition engine 402 that has been specifically trained to recognizespeech from person 1 using a voice 1 model 404, as described above.Voice 1 model 404 may be applied to each of speech fragment 400A andspeech fragment 400B as they are received by the entity tracker 100.

Upon processing the speech fragments, the entity tracker 100 outputs aprediction of the likelihood that each speech fragment corresponds toperson 1. As shown, for speech fragment 400A, the entity tracker outputsa person 1 identification 404A with a 90% confidence value, indicatingthat the speech fragment likely includes speech from person 1. Forspeech fragment 400B, the entity tracker outputs a person 1identification 404B with a 15% confidence value, indicating that speechfragment 400B likely does not include speech from person 1.

In some examples, an entity tracker 100 may be configured to identifybackground noise present in an environment, and use audio processingtechniques to subtract such background noise from received audio data.For example, a particular device in a person's home may be playingbackground audio, such as music or television/movie dialogue. Variousmicrophone-equipped devices in the person's home may record such audio.Where such microphone-equipped devices include the intelligent assistantsystem 20 and/or provide audio data to the entity tracker 100, suchbackground audio may compromise the ability of the system to identify,interpret and/or respond to human questions or commands.

Accordingly and in some examples, the device playing the backgroundaudio and/or another microphone-equipped device recording the backgroundaudio may send the captured audio signal to the entity tracker 100. Inthis manner, the entity tracker 100 may subtract the background audiofrom the audio signal received from the microphone-equipped devices. Insome examples, the subtraction of the background audio signal from therecorded audio data may be performed by the device(s) that capture theaudio data, or by associated audio-processing components, prior tosending the audio data to the entity tracker 100.

Additionally or alternatively, devices and/or the entity tracker 100 maybe trained to recognize particular sources of background noise (e.g.,from an air vent or refrigerator), and automatically ignore waveformscorresponding to such noise in recorded audio. In some examples, anentity tracker 100 may include one or more audio-recognition modelstrained specifically to recognize background noise. For example, audiofrom various noise databases may be run through unsupervised learningalgorithms in order to more consistently recognize such noise. Byallowing the entity tracker 100 to recognize irrelevant backgroundnoise, the ability of the entity tracker to recognize relevant humanspeech and other sounds may be improved.

With reference now to FIG. 11, in some cases the entity tracker 100 maydetermine that a change has occurred in the active participant (i.e.,the person currently speaking) in a conversation between two or morepeople. The entity tracker 100 also may determine at what point in timesuch a change occurred. This may be done in a variety of ways. In oneexample, a segment of recorded audio containing a speech fragment may betime-divided into two or more subframes, with each subframe containing adifferent sub-fragment of the recorded speech fragment. In the exampleof FIG. 11, speech fragment 500 may be time-divided into two or moresub-fragments, such as sub-fragment 1 502A and sub-fragment 2 502B. Invarious examples, the sub-fragments may have the same temporal length ordifferent temporal lengths. Where the sub-fragments have the sametemporal length, such length may be one second, two seconds, 10 secondsor any suitable length of time. Other examples of sub-fragments havingdifferent temporal lengths are described in more detail below.

Each sub-fragment of speech may be used to train a separate voice model,such that the trained voice model may be used to specifically recognizespeech from whichever person was speaking during that subframe. In FIG.11, sub-fragment 502A is used to train voice model 1 504A, whilesub-fragment 502B is used to train voice model 2 504B.

Once the sub-fragments have been used to train voice models, thesub-fragments may be cross-evaluated with different voice models. Thisis schematically shown in FIG. 12, in which voice model 2 504B, whichwas trained using sub-fragment 2 502B, is applied to sub-fragment 1502A. Similarly, voice model 1 504A, which was trained usingsub-fragment 1 502A, is applied to sub-fragment 2 502B.

During cross-evaluation, a prediction with a high confidence value willbe generated if the person speaking during the tested sub-fragment isthe same as the person speaking during the sub-fragment with which thevoice model was trained. Accordingly, if both voice models result inpredictions of the speaker identity with relatively high confidencevalues, then it is likely that the same person was speaking during bothsub-fragments, and the active participant in the conversation did notchange during the speech fragments. If both voice models result inpredictions of the speaker identity with relatively low confidence, thenit is likely that the active conversation participant changed at somepoint during the speech fragment. This possibility is illustrated inFIG. 12, in which voice model 2 504B outputs person identification 506Awith a confidence value of 30%, and voice model 1 504A outputs personidentification 506B with a confidence value of 25%. As both voice modelshave relatively low confidence in their predictions, it is likely thatdifferent people were speaking in each of sub-fragment 502A andsub-fragment 502B. It follows that it is likely that the activeconversation participant changed at some point between sub-fragment 1502A and sub-fragment 2 502B.

In some examples, the first confidence value associated with the firstsub-fragment and the second confidence value associated with the secondsub-fragment may be compared to a predetermined threshold to determineif the current speaker changed during the sub-fragments. In one example,a determination that the speaker of the first sub-fragment is thespeaker of the second sub-fragment may be made where both the firstconfidence value and the second confidence value exceed a predeterminedthreshold. The predetermined threshold confidence value may be 50%, 75%or other suitable value.

The predetermined threshold confidence value may be selected based onone or more of a variety of factors, such as the number of differentpersons detected in the current environment, an amount of background(non-speech) noise present, an estimated level of importance indetermining whether a speaker change occurs, context information 110,etc. In other examples, a determination that the speaker of the firstsub-fragment is the speaker of the second sub-fragment may be made wherejust one of the first confidence value and the second confidence valueexceed the predetermined threshold.

In some examples, a determination that the speaker of the firstsub-fragment is not the speaker of the second sub-fragment may be madewhere the first confidence value and the second confidence value areless than or equal to a predetermined threshold. In these examples, thepredetermined threshold confidence value may be 50%, 25% or othersuitable value. The predetermined threshold confidence value may beselected based on one or more of a variety of factors, such as thenumber of different persons in the current environment, an amount ofbackground (non-speech) noise present, a level of importance attributedto determining whether a speaker change occurs, context information 110,etc. In other examples, a determination that the speaker of the firstsub-fragment is not the speaker of the second sub-fragment may be madewhere just one of the first confidence value and the second confidencevalue is less than or equal to the predetermined threshold.

In some examples, the first confidence value associated with the firstsub-fragment and the second confidence value associated with the secondsub-fragment may be averaged and compared to a predetermined thresholdto determine if the speaker changed between the sub-fragments. In oneexample, a determination that the speaker of the first sub-fragment isthe speaker of the second sub-fragment may be made where the average ofthe first confidence value and the second confidence value exceed apredetermined threshold. The predetermined threshold confidence valuemay be 50%, 75% or other suitable value. In other examples, adetermination that the speaker of the first sub-fragment is not thespeaker of the second sub-fragment may be made where the average of thefirst confidence value and the second confidence value is less than orequal to a predetermined threshold. In these examples, the predeterminedthreshold confidence value may be 50%, 25% or other suitable value.

In some examples, the voice models may be generated from a universalbackground model. For example, a universal background model may comprisea speaker-independent Gaussian Mixture Model (GMM) that is trained withspeech samples from a large collection of speakers. For a givensub-fragment of speech, a plurality of acoustic features may beextracted and utilized to adapt the universal background model to moreclosely approximate the speaker of the particular sub-fragment. In oneexample, 100 acoustic features may be extracted from each sub-fragmentand utilized accordingly. In other examples, any suitable number offeatures may be extracted and utilized. In various examples, individualvoice models may be formulated using one or more statistical modelingtechniques, such as a maximum a posteriori probability (MAP) estimation.

In some examples, the system may update the universal background modelbased on determining whether a speaker change occurred. In one exampleand with reference to the example discussed above, where the systemdetermines that the speaker of the first sub-fragment is the same as thespeaker of the second sub-fragment, the universal background model maybe updated to an updated universal background model using the firstsub-fragment and the second sub-fragment. In this manner, by determiningthat a speaker change did not occur, the system may further refine theuniversal background model using multiple sub-fragments of speech fromthe same speaker.

In some examples, the system may subsequently utilize this updateduniversal background model to generate a third voice model. For example,after analyzing the first sub-fragment and the second sub-fragment todetermine that a speaker change did not occur, the system may captureanother sub-fragment of speech. In some examples, the other sub-fragmentof speech may be temporally contiguous with the second sub-fragment ofspeech (e.g., immediately following the second sub-fragment).

In one example, the other sub-fragment of speech may be used to trainthe updated universal background model to yield a third voice model. Asdescribed above regarding analyzing the first and second sub-fragments,the second sub-fragment may be analyzed with this third voice model toyield a third confidence value. Correspondingly, the third sub-fragmentis analyzed with the second voice model to yield a fourth confidencevalue. The third and fourth confidence values are then used to determineif the speaker of the second sub-fragment is the speaker of the thirdsub-fragment as described above.

In various examples and as noted above, the sub-fragments may have thesame temporal length or different temporal lengths. In some examples andwith reference to the examples of first and second sub-fragmentsdiscussed above, the first sub-fragment and the second sub-fragment mayhave unequal temporal lengths. With reference now to FIG. 24, in oneexample a portion of a speech fragment 360 received by the intelligentdigital assistant system 20 is schematically illustrated. In thisexample and as described in more detail below, the temporal lengths ofcontiguous sub-fragments may be adjusted and coordinated with theanalysis of the sub-fragments to reduce delay in identifying a speakerchange.

In the example of FIG. 24, the first sub-fragment 1 has a temporallength T1 and the second sub-fragment 2 has a shorter temporal lengthT2. Another sub-fragment 3 having temporal length T1 follows the secondsub-fragment 2, and another sub-fragment 4 having temporal length T2follows sub-fragment 3. As noted above, the first sub-fragment 1 may beused to train a first voice model and the second sub-fragment 2 may beused to train a second voice model. The first sub-fragment 1 iscross-evaluated with the second voice model to yield a first confidencevalue. The second sub-fragment 2 is cross-evaluated with the first voicemodel to yield a second confidence value. Based at least on the firstconfidence value and the second confidence value, the system determinesif the speaker of the first sub-fragment 1 is the same speaker of thesecond sub-fragment 2.

In this example, the cross-evaluations of sub-fragment 1 andsub-fragment 2 may be performed beginning at time T0. A total sampletime T3 for the two sub-fragments may be defined as a time T3=T1+T2.Because the second sub-fragment 2 is temporally shorter than the firstsub-fragment 1, the boundary 364 between the two fragments is closer totime T0 as compared to other configurations using a total sample time ofT3 in which the temporal lengths of the sub-fragments are equal, orwhere T2 is longer than T1, for example.

Accordingly, in the present example where T2 is shorter than T1, andwhere analysis of the confidence values indicates that a speaker changehas occurred, the occurrence of the speaker change is closer to T0(e.g., more recent in relation to the current time T0) than in otherconfigurations in which the total sample time is T3 and the temporallengths of the sub-fragments are equal, or where T2 is longer than T1,for example. It follows that the system's response to determining such aspeaker change can be executed more quickly in relation to theoccurrence of the speaker change. In this manner and in these examples,system actions and/or messages responsive to the speaker change mayoccur with less latency after the occurrence of the speaker change.

In different examples the magnitudes of T1 and T2 may be varied toaccommodate different use cases and a variety of system factors andconditions, such as processor loads, estimated importance of the currentuser input, etc. For example, T1 may be approximately 9 seconds while T2is approximately 3 seconds, or T1 may be approximately 8 seconds whileT2 is approximately 2 seconds. In different examples, the temporallength of T2 may be reduced to correspondingly reduce the temporal gapbetween the occurrence of a speaker change and the system'sdetermination of such change.

As the temporal length of T2 is reduced, the likelihood of identifying aspeaker change via cross-evaluation of sub-fragment 1 and sub-fragment 2is also reduced. Accordingly, the magnitudes of T1 and T2 may bedetermined and based at least in part on context information 110, suchas entity identity, entity position, and entity status informationreceived from the entity tracker 100 and described above.

As described above and with reference to FIG. 2, the parser 40 analyzestext and confidence values received from the voice listener 30 todetermine an intent of the user in speaking the received utterance. Theparser 40 may translate the natural language text received from thevoice listener 30 into code or data structures that represent a user'sintention underlying the natural language. The intent handler 50receives data representing user intentions from the parser 40, and insome examples may resolve ambiguous and missing information to generatecommitments.

Commitment engine 60 stores commitments from the intent handler 50. At acontextually appropriate time, the commitment engine may deliver one ormore messages and/or execute one or more actions that are associatedwith one or more commitments. Commitment engine 60 may store messages ina message queue 62 or cause one or more output devices 70 to generateoutput.

In some examples, and based at least on determining that the speaker ofthe first sub-fragment is the speaker of the second sub-fragment, theintelligent digital assistant system 20 may utilize at least the firstsub-fragment and the second sub-fragment to determine a user intent ofthe speaker. In this manner, by determining that the same person voicedboth the first sub-fragment and the second sub-fragment, bothsub-fragments may be utilized in determining the user intent of theperson speaking the current utterance. In addition to thesesub-fragments, other sub-fragments of the current utterance and/or otherutterances that are attributed to the same person may be used todetermine the user intent of the speaker.

In other examples, and based at least on determining that the speaker ofthe first sub-fragment is not the speaker of the second sub-fragment,the intelligent digital assistant system 20 may utilize the firstsub-fragment and forego utilizing the second sub-fragment to determine auser intent of the speaker of the first sub-fragment. In these examples,because the system determined that a different person voiced the secondsub-fragment, this sub-fragment may be ignored in determining the userintent of the speaker of the first sub-fragment. In addition to thefirst sub-fragment, other sub-fragments of the current utterance and/orother utterances that are attributed to the same person may be used todetermine the user intent of the speaker.

With reference now to FIGS. 25A and 25B, a flow chart of a method 1000for identifying a speaker change according to examples of the presentdisclosure is provided. The following description of method 1000 isprovided with reference to the software and hardware componentsdescribed herein. It will be appreciated that method 1000 also may beperformed in other contexts using other suitable hardware and softwarecomponents.

Additionally, while the blocks of method 1000 are described andillustrated in a particular sequence, in different examples the order ofexecution may vary. In some examples one or more of the blocks may notbe performed. In some examples, context information 110 from the entitytracker 100 may be utilized to determine an order of execution and/orwhich block to execute next.

With reference to FIG. 25A, at 1004 the method 1000 may include,receiving audio input comprising a speech fragment. At 1008 the method1000 may include training a first voice model with a first sub-fragmentfrom the speech fragment. At 1012 the method 1000 may include training asecond voice model with a second sub-fragment from the speech fragment.At 1016 the method 1000 may include analyzing the first sub-fragmentwith the second voice model to yield a first confidence value.

At 1020 the method 1000 may include analyzing the second sub-fragmentwith the first voice model to yield a second confidence value. At 1024the method 1000 may include, based at least on the first confidencevalue and the second confidence value, determining if a speaker of thefirst sub-fragment is the speaker of the second sub-fragment. At 1028the method 1000 may include, based at least on determining that thespeaker of the first sub-fragment is the speaker of the secondsub-fragment, utilizing at least the first sub-fragment and the secondsub-fragment to determine a user intent of the speaker.

At 1032 the method 1000 may include, based at least on determining thatthe speaker of the first sub-fragment is not the speaker of the secondsub-fragment, utilizing at least the first sub-fragment and foregoingutilizing the second sub-fragment to determine a user intent of thespeaker of the first sub-fragment. At 1036 the method 1000 may includegenerating the first voice model and second voice model from a universalbackground model. At 1040 the method 1000 may include, based at least ondetermining that the speaker of the first sub-fragment is the speaker ofthe second sub-fragment, updating the universal background model to anupdated universal background model using the first sub-fragment and thesecond sub-fragment.

With reference now to FIG. 25B, at 1044 the method 1000 may includegenerating a third voice model from the updated universal backgroundmodel by training the updated universal background model with anothersub-fragment of speech. At 1048 the method 1000 may include, wherein thefirst sub-fragment and the second sub-fragment have unequal temporallengths. At 1052 the method 1000 may include, based at least on thefirst confidence value and the second confidence value exceeding apredetermined threshold, determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment.

At 1056 the method 1000 may include, based at least on the firstconfidence value and the second confidence value being less than orequal to a predetermined threshold, determining that the speaker of thefirst sub-fragment is not the speaker of the second sub-fragment. At1060 the method 1000 may include computing an average of the firstconfidence value and the second confidence value. At 1064 the method1000 may include, if the average exceeds a predetermined threshold, thendetermining that the speaker of the first sub-fragment is the speaker ofthe second sub-fragment.

It will be appreciated that method 1000 is provided by way of exampleand is not meant to be limiting. Therefore, it is to be understood thatmethod 1000 may include additional and/or alternative steps relative tothose illustrated in FIGS. 25A and 25B. Further, it is to be understoodthat method 1000 may be performed in any suitable order. Further still,it is to be understood that one or more steps may be omitted from method1000 without departing from the scope of this disclosure.

In some examples, and depending on the sensors and processing methodsused by the entity tracker 100, tracking and identification of entitiesin an environment can be time-consuming and resource-intensive.Accordingly, the entity tracker 100 may use a variety of techniques toselectively choose when resource-intensive processing should beutilized. In this manner, the efficiency of the entity tracker 100 maybe improved without compromising its corresponding functionality.

As an example, a variety of image processing techniques may be used toaccount for variable lighting conditions in an environment. In someexamples, and depending on the brightness/darkness levels in a room, anentity tracker 100 can perform contrast adjustment and/or other imageprocessing techniques in order to more clearly track and identifyentities in the room. Such techniques, however, may require significantprocessing and computer resources. Accordingly and to conserve suchresources, additional context information 110 may be evaluated todetermine whether to utilize such techniques.

For example, where a room is dark and context information 110 with highconfidence values indicates the room is empty, the entity tracker 100may forego computationally-intensive image processing techniques infavor of conserving resources. In another example, where another sensorin the room detects that a person is likely present (e.g., a microphonerecords the person's voice), the entity tracker 100 may authorize theuse of computationally-intensive image processing in an attempt toobtain an image that can be used to identify the person's face. Inanother example, an entity tracker 100 may reduce the sampling frequencyof any sensors monitoring an environment in which no entities ofinterest are currently present. Thereafter, the entity tracker 100 mayincrease the sampling frequency of one or more sensors as needed, suchas when the presence of an entity of interest is indicated with aconfidence value exceeding a predetermined threshold.

Another process which can require significant computer resources isfacial recognition using high-resolution images. In some examples, uponestablishing a positive identification of a person usingfacial-recognition techniques, the entity tracker 100 may switch to lessresource-intensive identification methods in order to continue trackingthe person. As an example, upon detecting that a new person has entereda room, the entity tracker 100 may capture a high-resolution image ofthe person's face. The entity tracker 100 may utilize this image toperform relatively resource-intensive facial recognition in order todefinitively identify the person.

After initial identification of the person, the entity tracker 100 mayuse less resource-intensive techniques in order to continue tracking theperson while conserving computing resources. For example, the entitytracker 100 may use lower-resolution cameras to track the person basedon the general shape of their body, their gait (e.g., by evaluatingangles formed between different joints as the person walks), theirclothing (e.g., tracking patches of color known to correspond to theperson's clothing), etc. In some examples, and to periodically confirmits initial identification of the person is still accurate, the entitytracker 100 may perform facial recognition intermittently after theinitial identification. In general and depending on the particularcontext, the entity tracker 100 may use any of a variety ofidentification techniques in order to intelligently manage bothconservation of computing resources and identification and tracking ofentities.

As noted above, the commitment engine 60 stores commitments receivedfrom the intent handler 50. Also as described above, the commitmentengine 60 may utilize one or more cost functions to determine one ormore costs associated with executing or not executing a commitment and,in some examples, with outputting or not outputting a message to theuser. As described in more detail below, in some examples one or moremessages may be added to a message queue.

With reference now to FIG. 13, a schematic illustration of an examplecost function 620 usable with commitment engine 60 is provided. Thecommitment engine 60 includes commitment storage 626 configured to storeone or more commitments 614. The commitment 614 shown in FIG. 13includes a trigger condition 616 and a message/action 618. In someexamples, a commitment may comprise a set of 0 or more triggers and aset of 1 or more messages/actions (such as convey a message to a user,turn on the lights, play music, etc.). In some examples, themessage/action 618 may comprise conveying a message 619 stored in amessage queue 62 as output 670 via one or more output devices 70. Insome examples, the message/action 618 may comprise executing one or moreadditional instructions that may include interfacing with one or moreother devices or programs.

The commitment engine 60 is configured to receive context information110 via entity tracker 100 and/or directly from one or more sensors 22.In various examples, the context information 110 may or may not satisfya trigger condition, such as trigger condition 616. The contextinformation 110 may include entity data 630 from the entity tracker 100.The context information 110 may further include time data 650 and/or auser input 680. The entity data 630, time data 650, and user input 680are described in greater detail below.

When the commitment engine 60 receives context information 110 thatsatisfies the trigger condition 616 of the commitment 614, thecommitment engine 60 may apply a cost function 620 to the message/action618. Where the message/action 618 comprises a message, the cost function620 is configured to determine whether to output the message associatedwith the commitment 614 to one or more users. Similarly, where themessage/action 618 comprises an action, the cost function 620 isconfigured to determine whether to perform the action based at least inpart on the context information 110. When the cost function 620determines that the commitment 614 is of high enough importance, thecommitment engine 60 may output the message and/or perform the action ofmessage/action 618 as an output 670. The output 670 may be conveyed toone or more output devices 70. For example, the output 670 may comprisea verbal message that is broadcast by a speaker of a user's smartphoneand/or one or more other speakers in the user's environment, such as astandalone speaker device, a television speaker, etc. In some examples,the output 670 may comprise controlling one or more other devices, suchas turning on lights, playing music via a media program, etc.

The cost function 620 may determine whether to output a message 619 bycalculating a cost of notification 622 and a cost of non-notification624. If the cost of non-notification 624 is determined to be higher thanthe cost of notification 622, the commitment engine 60 may output themessage 619. In some examples, the cost of notification 622 and the costof non-notification 624 may be determined at least in part using one ormore machine learning algorithms.

In some examples, the cost function 620 may determine the cost ofnotification 622 and the cost of non-notification 624 based at least inpart on entity data 630 received from the entity tracker 100 andincluded in the context information 110. As explained above, the entitydata 630 may include entity identity data 112, entity position data 114,and entity status data 116. The entity identity data 112, entityposition data 114, and entity status data 116 may each include at leastone list of users, locations, and activities respectively. Each user,location, and activity included in the entity identity data 112, entityposition data 114, and entity status data 116 may have an associatedestimate of a probability that that user, location, or activity wascorrectly identified. Such probability estimates may be utilized by thecost function 620 in determining corresponding costs of notification 622and costs of non-notification 624. In some examples, the contextinformation 110 may include entity identity data 112, entity positiondata 114, and entity status data 116 for one or more users or otherentities detected simultaneously.

The commitment engine 60 also may be configured to receive time data650. The time data 650 may include a time 652 and/or date 654. The timedata 650 also may include at least one time interval 656 elapsed since acomputing device performed some task. For example, the time data 650 mayinclude at least one time interval 656 elapsed since a computing deviceproduced a particular output or received a particular input. Forexample, a user may set a time interval 656 on an oven timer whilebaking bread, and the commitment engine 60 may receive contextinformation 110 that satisfies the trigger condition 616 when the timeinterval 656 elapses and the oven timer buzzes. In response, the costfunction 620 may be applied to a related commitment 614 to determinewhether to output a message 619 that the bread should be removed fromthe oven, and/or perform an action to turn off the oven.

As another example, the time data 650 may include data indicating when acomputing device most recently produced an output 670 notifying a userthat the user should do laundry. The message queue 62 may store arelated message 619 reminding the user to do his laundry. As the amountof time increases since the message was last given, as indicated by thetime data 650, the cost function 620 may gradually increase the cost ofnon-notification 624. When the laundry notice is given, the costfunction 620 may decrease the cost of non-notification 624 of themessage 619. In some examples, the cost function 620 may determine thecost of notification 622 and the cost of non-notification 624 based atleast in part on the time data 650.

The cost function 620 may determine the cost of notification 622 and thecost of non-notification 624 based at least in part on one or more userinputs 680. For example, a user may provide a user input 680 thatincreases the cost of non-notification 624 for a notification the userconsiders particularly important. The user may, in one example, providea user input 680 to increase the cost of non-notification 624 for a jobinterview compared to a default cost of non-notification 624 for a lunchmeeting with a friend.

The commitment engine 60 may determine an output type for the output670. The determination of the output type may be made based on theentity data 630 and/or time data 650. For example, the commitment engine60 may determine, based on user location data, that a user is not in alocation at which the user would be able to view a display screen. Thecommitment engine 60 may therefore generate an output 670 that isconveyed to a speaker instead of the screen. In addition, some outputtypes may have costs of notification 622 different from those of otheroutput types. For example, a visual display may have a lower cost ofnotification 622 than a sound output, since a visual output may be lesslikely to interrupt a conversation.

In one example, the commitment engine 60 may receive context information110 that satisfies the trigger condition 616 for a commitment 614 withthe message “John has a meeting in 15 minutes.” A cost function 620 thenmay be applied to the commitment 614. The cost function 620 may useentity data 630 to determine a cost of notification 622 and a cost ofnon-notification 624. The cost of notification 622 and the cost ofnon-notification 624 may be determined based at least in part on factorssuch as how important the message is to John, how receptive John is toreceiving the message, and whether John is in a location that may berelated to the message. For example, the entity data 630 may includeentity identity data 112 and entity position data 114 that indicate thatJohn is not currently in a location in which he could perceive an output670 from the output device 672. As a result, the cost ofnon-notification 624 may be very small. In another example, based onentity data 630, the commitment engine 60 may determine that the cost ofnotification 622 is higher when John is making a telephone call thanwhen John is reading a newspaper. In another example, where the messagecontent includes medical information about John's newborn baby, the costof non-notification 624 may be determined to be high.

In another example, the commitment engine 60 may receive contextinformation 110 indicating that a baby is in a location near a user'sswimming pool. This context information 110 may satisfy a triggercondition 616 for a commitment 614 corresponding to the situation inwhich a baby is near the user's pool. The commitment 614 may include amessage/action 618 to broadcast an urgent message to a user that a babyis near the pool. A cost function 620 may be applied to a commitment614. Based at least in part on the trigger condition 616 of thecommitment 614 relating to a potentially serious safety situationinvolving a baby, the cost function 620 may determine that thecommitment 614 has a very high cost of non-notification 624.

Continuing with this example, based on entity data 630 including useractivity data, the commitment engine 60 may determine that the user iscurrently making a telephone call. The commitment engine 60 also mayaccess profile data of the user indicating that the user stronglyprefers to avoid interruptions when he is talking on the phone. As aresult, the commitment engine 60 may determine that the cost ofnotification 622 is also high. In this example, given that the messagerelates to a safety situation involving a baby, the commitment engine 60may determine that the cost of non-notification 624 is higher than thecost of notification 622. Accordingly, the commitment engine 60 conveysthe urgent message 619 as an output 670 to be output by the outputdevice 672 to the user.

In another example, commitment engine 60 may receive context information110 that triggers the trigger condition 616 for a commitment 614 withthe message “John has a meeting in 15 minutes.” Using entity data 630,the commitment engine also may determine that John is currently making atelephone call. The commitment engine 60 may determine that sinceoutputting a message notifying John of the commitment 614 on the outputdevice 672 would interrupt John's telephone call, the cost ofnotification 622 is greater than the cost of non-notification 624. Thus,the commitment engine 60 may not convey the message to the output device672 as output 670.

As the time of John's meeting approaches, the commitment engine 60 mayincrease the cost of non-notification 624 based on time data 650. Forexample, the commitment engine 60 may determine that John is fiveminutes away from the location of the meeting. When the time data 650indicates that the meeting will begin in six minutes, the cost ofnon-notification 624 may be high enough that the commitment engine 60conveys the message 619 to the output device 672 even though John isstill making the telephone call.

A flowchart of a method 700 for determining a cost of notification and acost of non-notification of a message is shown in FIG. 14. The followingdescription of method 700 is provided with reference to the software andhardware components described herein. It will be appreciated that method700 also may be performed in other contexts using other suitablehardware and software components.

At 702 the method 700 may include receiving context information.Receiving the context information may include receiving entity data,time data, and/or a user input. At 704 the method 700 may includereceiving entity data including entity identity, position, and/or statusdata. At 706 the method 700 may include receiving time data. The timedata may include a time and/or date. The time data may further includeat least one time interval. At 708 the method 700 may include receivinga user input.

At 710 the method 700 may include detecting that a trigger condition hasoccurred, based at least in part on the context information. The triggercondition may be associated with a commitment.

At 712 the method 700 may include determining a cost of notificationthat is associated with outputting the message on an output device. At714 the method 700 may include determining a cost of non-notificationthat is associated with not outputting the message. In some examples,determining the cost of notification and the cost of non-notificationmay be based at least in part on the entity data, time data, and/or userinput. In some examples, the cost of notification and cost ofnon-notification may be determined at least in part using a machinelearning algorithm.

At 716 the method 700 may include comparing the cost of notification tothe cost of non-notification. At 718 the method 700 may include, if thecost of non-notification exceeds the cost of notification, conveying themessage to be output on the output device. At 720 the method 700 mayinclude, if the cost of non-notification does not exceed the cost ofnotification, refraining from conveying the message to the outputdevice.

It will be appreciated that method 700 is provided by way of example andis not meant to be limiting. Therefore, it is to be understood thatmethod 700 may include additional and/or alternative steps relative tothose illustrated in FIG. 14. Further, it is to be understood thatmethod 700 may be performed in any suitable order. Further still, it isto be understood that one or more steps may be omitted from method 700without departing from the scope of this disclosure.

In some examples, one or more cost functions may be used to determineand/or adjust a sampling rate of one or more sensors. In some use casescenarios, sampling and analyzing an environment with sensors may be acomputationally intensive task. As explained above, a cost function maydetermine a cost of executing and/or not executing an action (such ascommunicating a message). Using such determinations, the system mayincrease or decrease the sample rate of one or more sensors that providedata related to the action. For example, where the action comprisesnotifying a user via outputting a message on an output device, a sensorrate may be increased or decreased based at least in part on themagnitude of the cost associated with refraining from outputting themessage (e.g., non-notification).

In some examples, one or more cost functions may be used to customize atype and/or manner of delivery of a notification. In one example, a costfunction may determine that a cost of non-notification of a message maybe very high. For example, a user may establish a rule with the systemthat any messages containing the word “baby” are assigned a highest,critical importance status. Accordingly, where a message includes theword “baby”, a cost function may determine that the cost ofnon-notification is very high, and correspondingly may broadcast themessage at a high volume via all available speakers in a user's home.

With reference now to FIG. 15, a schematic representation of a thresholdupdating module 810 according to examples of the present disclosure isprovided. In different examples, threshold updating module 810 may be astandalone module in intelligent assistant system 20, or may be acomponent of the entity tracker 100, parser 40, or commitment engine 60.In some examples, the threshold updating module 810 may be configured tomodify threshold data 820 that may be used to parse entity data 832. Thethreshold data 820 may include an entity identification threshold 822,an entity position/location threshold 824, and an entity statusthreshold 826. Each of these thresholds may be defined as a probability.When an entity identity, location, or status is determined to have adetection probability that exceeds the threshold probability for thatentity identity, location, or status, a detection of that entityidentity, location, or status may be indicated and/or recorded.

The threshold data 820 may be updated by the threshold updating module810 to produce updated threshold data 850. The updated threshold data850 may include an updated entity identification threshold 852, anupdated entity location threshold 854, and an updated entity statusthreshold 856. The threshold updating module 810 may update thethreshold data 820 based on a cost function 830, entity data 832, timedata 834, and/or user input 836. In some examples, the cost function830, entity data 832, and time data 834 may be the cost function 620,entity data 630, and time data 650 of FIG. 13.

In some examples, the threshold updating module 810 may update thethreshold data 820 based on a modification of the cost function 830. Asdescribed above, the cost function 830 may be configured to determine acost of notification and a cost of non-notification for messages thatmay be conveyed for output. In some examples, the modification to thecost function 830 may be made in response to a user input 836. Forexample, a sleep-deprived user may enter an input that increases thecost of notification when that user is determined to be sleeping. As aresult, the threshold updating module 810 may decrease a user statusthreshold 826 for determining that the user is sleeping. In someexamples, the user may enter an input 836 that directly updates thethreshold data 820.

The threshold updating module 810 may also update the threshold data 820based on entity data 832. As noted above, the entity data 832 mayinclude entity identification data, entity location data, and/or entitystatus or activity data. In one example, threshold data 820 may includean entity identification threshold 822 and an entity location threshold824 for detecting that both a child and an adult are simultaneously inproximity to a hot stove. The threshold updating module 810 may receiveentity data 832 indicating that a child is alone in proximity to the hotstove. In response, the threshold updating module 810 may revise thecorresponding entity identification threshold 822 and entity locationthreshold 824 to be lower.

In some examples, the threshold updating module 810 may also update thethreshold data 820 based on time data 834. The time data may include atime, date, and/or at least one time interval that has elapsed since aparticular input has been outputted, or since a particular input hasbeen received. For example, the entity status threshold 826 for sleepingmay be lowered when the time data 834 indicates that it is nighttime.

A flowchart of a method 900 for updating threshold data is provided inFIG. 16. The following description of method 900 is provided withreference to the software and hardware components described herein. Itwill be appreciated that method 900 also may be performed in othercontexts using other suitable hardware and software components.

At 902 the method 900 may include receiving a set of threshold data. Thethreshold data may include one or more probability thresholds abovewhich a detection of a user, user location, or user activity may beregistered. At 904 the method 900 may include receiving a modificationto a cost function. At 906 the method 900 may include receiving entitydata, which may include entity identification data, entityposition/location data, and/or entity status data.

At 908 the method 900 may include receiving time data, which may includea time, a date, and/or at least one time interval elapsed since aparticular output was produced or a particular input was received. At910 the method 900 may include receiving a user input. At 912 the method900 may include producing updated threshold data by modifying thereceived threshold data based at least in part on the cost functionmodification, entity data, time data, and/or user input.

It will be appreciated that method 900 is provided by way of example andis not meant to be limiting. Therefore, it is to be understood thatmethod 900 may include additional and/or alternative steps relative tothose illustrated in FIG. 16. Further, it is to be understood thatmethod 900 may be performed in any suitable order. Further still, it isto be understood that one or more steps may be omitted from method 900without departing from the scope of this disclosure.

In some examples, intelligent assistant system 20 may train users tointeract with the system in ways that minimize ambiguities. For example,the system may customize responses, queries, and other interactions witha particular user in a manner that trains the user to speak to thesystem using words or phrases that more clearly convey a user's intent.In this manner, follow-up queries from the system to clarify ambiguitiesmay be minimized. In one example, where a user has two contacts namedFred (Jones and Smith), and the user frequently tells the system, “CallFred”, the system may offer the suggestion, “When you′d like to call oneof your contacts Fred, it would be helpful to use his last name too.”

In some examples, intelligent assistant system 20 may customize a userinterface to provide additional opportunities for collecting user datathat may be used to enhance user experiences. In some examples, suchdata may be utilized with machine learning techniques to learn userpreferences and make predictions from such information. In one example,where a user utilizes a calendar application that provides reminders,the system may provide a user interface, or may modify an existing userinterface, to gather useful data about the user. For example, whenproviding a reminder the calendar application may provide two defaultoptions of Dismiss and Snooze, with the Snooze period selectable inseveral 5 minute increments.

In some examples, intelligent assistant system 20 may modify thereminder user interface to display two different Snooze selectors withdifferent labels—“Not now I'm busy” and “Not now, it's not important.”Accordingly, when a user selects one of these more detailed selectors,the system may learn about the user; namely, what activities, persons,types of meetings, etc., the user considers “not important” or make theuser “busy.” In this manner, such information helps the systemunderstand more about the user. As such data is gathered over time,machine learning techniques may be utilized to better understand userpreferences and other attributes. In other examples, many other types ofdata (image, audio, physiological, etc.) may be gathered in conjunctionwith providing customized user interface experiences to learn more abouta user.

With reference now to FIGS. 17-21, additional example implementations ofintelligent assistant system 20 in a single computing device and acrossmultiple computing devices are illustrated. Additional details regardingcomponents and computing aspects of computing devices illustrated inFIGS. 17-21 are described below with reference to FIG. 23.

FIG. 17 shows an example of an all-in-one computing device 160 in whichthe components implementing intelligent assistant system 20 are arrangedtogether in a standalone device. In some examples, all-in-one computingdevice 160 may be communicatively coupled to one or more other computingdevices 162 via a network 166. In some examples, all-in-one computingdevice 160 may be communicatively coupled to a data store 164 that maystore a variety of data, such as user profile data. All-in-one computingdevice 160 includes at least one sensor 22, voice listener 30, parser40, intent handler 50, commitment engine 60, entity tracker 100, and atleast one output device 70. Sensor(s) 22 include at least one microphoneto receive natural language inputs from a user. In some examples one ormore other types of sensor(s) 22 also may be included.

As described above, voice listener 30, parser 40, and intent handler 50work in concert to convert natural language inputs into commitments thatare executable by the all-in-one device 160. The commitment engine 60stores such commitments in a commitment storage 626. The entity tracker100 may provide context information to the commitment engine 60 and/orother modules. At a contextually appropriate time, the commitment engine60 may execute a commitment and provide output, such as audio signals,to output device(s) 70.

FIG. 18 shows an example implementation in which one or more remoteservices 170 perform the natural language processing functionality ofintelligent assistant system 20. In this example, voice listener 30,parser 40, intent handler 50, entity tracker 100 and commitment engine60 reside on one or more computing devices, such as one or more servers,that are remotely located from a cloud-supported user device A. Sensordata from one or more sensors 22 of the user device A is provided toremote service(s) 170 via a network. For example, audio data of a userspeaking may be captured by a microphone of user device A and providedto voice listener 30.

As described above, voice listener 30, parser 40, and intent handler 50cooperate to convert the audio data into commitments that are stored incommitment engine 60. At a contextually appropriate time, the commitmentengine 60 may execute a commitment and provide output, such as audiosignals, to one or more output device(s) 70 of the user device A.

FIG. 19 shows another example implementation in which one or more remoteservices 170 perform the natural language processing functionality ofintelligent assistant system 20. In this example, the one or more remoteservices 170 are communicatively coupled with a plurality of differentsensors 22 and output devices 70. In this example, the sensors includeindividual standalone sensors A and C, such as microphones, cameras,etc. The output devices include individual standalone output devices Band D, such as loudspeakers.

The one or more remote services 170 are also communicatively coupled toa device E that includes one or more sensors F and an output device G.Device E may take the form of a simple standalone device comprising amicrophone, speaker and network connectivity components. In otherexamples, device E may be a mobile phone, tablet computer, wall-mounteddisplay, or other suitable computing device. In some examples, device E,sensors A and C, and output devices B and D may be part of the samecloud-supported client. In other examples, any number of individualsensors and devices may be utilized with the one or more remote services170.

As described above, the one or more remote services 170 perform thenatural language processing functionality of intelligent assistantsystem 20. In some examples, one or more of the remote services 170 mayinclude all of the natural language processing modules of intelligentassistant system 20, as shown in the example of FIG. 18. In otherexamples, one or more remote services 170 may include less than all ofthe natural language processing modules, and may be communicativelycoupled to the other modules located at one or more other service(s). Inthe present example, and as described in more detail below, one or moreof the remote services 170 also may comprise a device selector 174 thatmay utilize sensor inputs to select output device B, D and/or G toreceive output from the commitment engine 60.

With reference now to FIG. 20, in some examples the intelligentassistant system 20 of the present disclosure may utilize deviceselector 174 to enable a user to communicate with another person whoselocation may be unknown to the user. In some examples, the system mayuse sensor data and/or corresponding context data to detect the presenceand determine the location of the other person. Upon receiving a requestfrom the user to speak to or locate the other person, the deviceselector 174 may select an appropriate output device for establishingcommunication between the user and the other person.

In the example use case of FIG. 20, one or more remote services 170implementing intelligent assistant system 20 are communicatively coupledwith a smartphone 190 and laptop 192. In one example, smartphone 190comprises multiple sensors A including a microphone, and an outputdevice A in the form of a speaker. The smartphone 190 may be locatedwith a user in the user's basement media room of her home. The laptopcomputer 192 comprises multiple sensors B including a microphone and awebcam, and an output device B in the form of a speaker. The laptop 192may be located in an upstairs bedroom of the home.

The user of the smartphone 190 may desire to communicate with herdaughter, but may not know her current location within the home. Thedaughter may be in the upstairs bedroom with two other friends. The usermay speak natural language inputs to indicate that she would like tocommunicate with her daughter. For example, the user may speak “Connectme to Sarah.” The microphone in the user's smartphone 190 may receivethe natural language input and send it to a remote service 170 forprocessing by the voice listener 30 and other components of intelligentassistant system 20 described above.

Upon determining the intent of the user, the commitment engine 60 mayrequest context information 110 from the entity tracker 100 thatincludes the location of the user's daughter Sarah. In response, theentity tracker 100 may utilize video data from the webcam of the laptop192 to identify Sarah in the field of view of the webcam. Entity tracker100 may use other context information to determine that the laptop 192,and thus daughter Sarah, are located in the upstairs bedroom.

Using this information, the device selector 174 may communicativelycouple the microphone and speaker of the user's smartphone 190 withmicrophone and speaker of laptop computer 192, and thereby allow theuser to talk with her daughter.

In other examples and as discussed above, one or more other types ofsensors and corresponding data may be used to locate a person or otherentity. Examples include solely audio data, combinations of video andaudio data, device log-in data, and other combinations of the foregoingand other sensor data.

In some examples, one or more functions of the intelligent assistantsystem 20 may be activated upon detection of one or more keywords thatare spoken by a user. For example, the phrase “Hey Computer” may be usedas a keyword phrase to activate one or more functions of the system.With reference now to FIG. 21, in one example one or more sensors 22 inthe form of microphones may receive audio data of a user speaking “Heycomputer, what time is the school board meeting tonight?” As describedabove, the voice listener 30 may process the audio data into text andconfidence value(s), and pass this information to the parser 40. Anattention activator 32 in parser 40 may identify the keyword phrase “Heycomputer” in the text. In response, the parser 40 may activate or modifyother components and functionality of the intelligent assistant system20. For example, the parser 40 may increase a sampling rate of a speechrecognition module to increase recognition accuracy of the user's speechthat is likely to follow.

As noted above, upon processing audio data of a user's natural languageinput, a commitment engine may provide output to one or more outputdevices, such as a speaker and/or a video display. In some examples, asingle device may include a microphone that captures a user's input,with such input provided to the intelligent assistant system 20, and aspeaker that receives and broadcasts a message generated by the systemin response to the input.

In some examples, a user may be in an environment with two or moremicrophones that may capture user speech and/or two or more speakersthat may broadcast a message generated by the system in response to thespeech. For example, a user may be in his media room with his mobilephone, laptop computer, tablet computer, and smart/connected television.Each of these devices may contain or be communicatively coupled with anintelligent assistant system 20.

A user may speak a keyword phrase that is captured by the microphones ofeach of the 4 devices. Accordingly, the corresponding message generatedby the intelligent assistant system 20 may be broadcast by the speakersin all 4 devices, which may be annoying to the user. As described inmore detail below, in some examples involving multiple sensors, outputdevices and/or other devices, the intelligent assistant system 20 may beconfigured to determine which of the multiple microphones to use forreceiving user speech and/or which of the multiple speakers to use forbroadcasting a corresponding message. In some examples and as describedbelow, an aggregator may evaluate and weigh a plurality of metrics todetermine which microphones and speakers to utilize.

With reference now to FIG. 22, an example implementation of sensor andoutput device selection in response to voice activation in amulti-device environment is provided. In this example, one or moreremote services 170 that implement the intelligent assistant system 20may receive audio data from three different microphones A, B and C ofthree different devices, such as a mobile phone 176, tablet computer 178and all-in-one intelligent assistant device 180.

A user in the vicinity of the three devices may speak a keyword phrase,such as “Hey Computer.” Each of the microphones A, B and C may captureaudio data of the user speaking this phrase and may send the audio datato voice listener 30. As described above, voice listener 30 may utilizespeech recognition technologies to translate spoken utterances intotext. Voice listener 30 also may assign confidence value(s) to thetranslated text. In some examples, the voice listener 30 may comprise akeyword detection algorithm configured to identify a keyword or keywordphrase in the translated text. The voice listener 30 may assign aconfidence value to text that indicates a likelihood that the text is akeyword or keyword phrase.

In some examples, an aggregator 182 may evaluate a plurality of metricsrelated to a plurality of user audio data streams that are received fromdifferent individual microphones and/or from different microphonearrays. As described in more detail below, the aggregator 182 mayutilize these metrics to select one of the audio data streams and itscorresponding microphone(s) for use in interacting with the user. Insome examples, the microphone(s) that is determined to be closest to theuser may be selected. In some examples, the microphone(s) that isdetermined to provide the highest quality audio data may be selected. Insome examples, the microphone(s) providing the highest quality audiodata may be determined to be the closest microphone(s) to the user, andtherefore may be selected.

When a microphone has been selected, the device selector 174 may selecta speaker associated with this microphone to output a response to theuser. For example, where the microphone is a component of a device thatincludes a speaker, this speaker may be selected. Where the microphoneis a standalone microphone, the aggregator 182 may select anotherspeaker in the vicinity of the user to output the response. In theexample of FIG. 22, the aggregator 182 is located on one of the remoteservices 170 that implements at least a portion of the intelligentassistant system 20. In other examples, the aggregator 182 may belocated on another computing device, such as in another cloud-basedservice.

In one use case example, the aggregator 182 may utilize 4 metrics toevaluate a user audio data stream that is received: (1) an amplitude(volume) of the received audio signal; (2) a signal-to-noise (S/N) ratioof the audio signal; (3) a keyword confidence value indicating alikelihood that the data stream contains a keyword or keyword phrase;and (4) a user identification confidence value indicating a likelihoodthat the speaker is a particular person.

In some examples, the amplitude and/or S/N values may be received withthe audio data stream. In other examples, amplitude and/or S/N valuesmay be determined by the voice listener 30 or other components of theintelligent assistant system 20. As described above, the keywordconfidence value may be determined by the voice listener 30. Also asdescribed above, the user identification confidence value may bedetermined by entity tracker 100. In some examples, the user speakingthe input may be identified by voice recognition as a known speaker oran unknown speaker, and assigned a corresponding level of confidence.

The S/N ratio may be calculated for the received audio input bycomparing a signal level of a user's voice to a level of backgroundnoise. In some examples the amplitude of the input may be used todetermine a proximity of the user to the corresponding microphone. Itwill be appreciated that the metrics discussed in the presentimplementations are provided as examples and are not meant to belimiting.

Each of the received audio data streams also may include a device IDthat identifies the particular device or standalone sensor that isproviding the data stream. In some examples, after receiving a first setof metrics from a first device or sensor, the aggregator 182 may pausefor a predetermined period of time to determine if one or more otherdevices/sensors also received the keyword or keyword phrase from thesame person as the user identified in the first set of metrics. Forexample, the aggregator 182 may pause for 0.5 seconds, 1.0 seconds, orany other period of time that does not create a negative user experiencefor the user.

In the present example and as shown in FIG. 22, the aggregator 182evaluates metrics for audio data streams received from the mobile phone176, tablet computer 178 and all-in-one intelligent assistant device180. For each device, the aggregator 182 may combine the 4 metrics intoa single selectability score, such as by averaging the 4 metrics. Insome examples and prior to combining, each of the metrics may beweighted by empirically-determined weights that reflect the accuracy ofa metric in predicting the device/microphone and corresponding audiodata stream that will provide the best user experience. By comparing theselectability scores of each of the devices/microphones and their datastreams, the aggregator 182 may identify and select the desireddevice/data stream.

In one example, for each of the 4 metrics, the aggregator 178 maycompare the scores of each device/microphone and correspondingly rankthe devices/microphone per metric. For example, the aggregator 178 maydetermine the following scores for the audio data stream received frommicrophone A of the mobile phone 176: 1) 90% (Amplitude); 2) 90% (S/N);3) 30% (Keyword confidence); 4) 90% (Speaker ID). Scores for the audiodata stream received from microphone B of the tablet computer 178 maybe: 1) 80% (Amplitude); 2) 80% (S/N); 3) 80% (Keyword confidence); 4)80% (Speaker ID). Scores for the audio data stream received from themicrophone C of the intelligent assistant device 180 may be: 1) 92%(Amplitude); 2) 88% (S/N); 3) 90% (Keyword confidence); 4) 92% (SpeakerID).

In this example, the rankings of the 3 devices for each of the 4 metricswould be as follows:

-   -   A. Amplitude—1. Intelligent assistant device; 2. Mobile        phone; 3. Tablet computer.    -   B. S/N Ratio—1. Mobile phone; 2. Intelligent assistant        device; 3. Tablet computer.    -   C. Keyword Confidence—1. Intelligent assistant device; 2. Tablet        computer; 3. Mobile phone.    -   D. Speaker ID—1. Intelligent assistant device; 2. Mobile        phone; 3. Tablet computer.

Each device may be awarded points based on its ranking in each metriccategory. For example, a first place ranking=1 point, second place=2points and third place=3 points. For each device, its points are totaledfor the 4 metrics and averaged. The aggregator 182 selects the device(and corresponding data stream) with the lowest average point total. Inthe present example, the final point totals and rankings are: 1.Intelligent assistant device=>1.25; 2. Mobile phone=>2.0; 3. Tabletcomputer=>2.75. Thus, the aggregator 178 selects the data stream fromthe intelligent assistant device 180 n for continued analysis by theintelligent assistant system 20. Additionally, and based on the aboveranking, the device selector 174 may select the intelligent assistantdevice 180 to receive the message(s) generated by commitment engine 60as a result of the analysis.

In some examples, upon selection by the aggregator 182 of theintelligent assistant device 180 as described above, the aggregator alsomay cause the other two devices to refrain from sending audio datastreams that are associated with the same speaker ID (i.e., person) thatis associated with the analyzed data stream. In this manner, where thesame user provides more natural language input after the initial input,only the selected intelligent assistant device 180 will provide thecorresponding audio data to the remote service(s) 170. In some examples,the other two devices may resume sending audio data streams when thesame person speaks the keyword or keyword phrase. In these cases, theabove-described selection process may be performed again to determinethe selected device.

In some examples and as noted above, prior to averaging the awardedpoints, each point award may be multiplied by an empirically-determinedweighted value that reflects the accuracy of a metric in predicting thedevice and corresponding audio data stream that will provide the bestuser experience. In some examples, one or more machine learningtechniques may be utilized to build models for computing the differentmetrics.

In some example implementations, the signal amplitude may stronglycorrelate to a user's distance from the microphone receiving the user'sspeech. The S/N ratio also may provide a good indicator of the user'sdistance from the microphone, as a lower noise value may correlate tothe user being closer to the microphone. Where the signal amplitude andS/N ratio of the signal are both relatively high, the speaker IDaccuracy may correspondingly benefit from the strong signal.

It will be appreciated that the methods and use cases described aboveare merely examples, and many variations are possible. For example, asubset of the above 4 metrics may be utilized to evaluate a user audiodata stream. In other examples, one or more additional metrics also maybe utilized.

In some examples, a user who has previously established a conversationwith the intelligent assistant system 20 via a selected device amongmultiple devices may have a brief pause before initiating a nextconversation with the same device. The system may compare the durationof the pause to a predetermined time period, and may consider thecomparison in selecting a device for the next conversation. For example,where the duration of the pause is less than the predetermined period,such as 5 seconds, the system may include the recently-establishedspeaker ID and the existence of the previous conversation in the devicedetermination analysis as a bias towards selecting the same device forthe next conversation.

The examples described above include recognition of an audible keywordto activate one or more functions of the intelligent assistant system.In some examples, functions of the system may be activated byrecognition of one or more other signals. Such signals may include, forexample, a user gesture captured by a camera, a user eye-gaze, and aface direction of the user.

In some examples, one or more of the above-described techniques fordevice selection may be utilized to automatically update the selecteddevice based on one or more factors. For example, where a user iscommunicating with the intelligent assistant system 20 via a firstdevice, as the user changes her location and moves farther away from thefirst device, the system may correspondingly change the selected deviceto a second device closer to the user's new location.

In some implementations, imaging data in addition to audio data from oneor more image sensors may be utilized to select a device. For example,context data 110 received from entity tracker 100 may include imagingdata that may be used to select a device. Examples of imaging data mayinclude video from an RGB camera, infrared images from an IR camera,depth images from a depth camera, thermal images from a thermal camera,etc. For example, an RGB camera may track a user's location within aroom. Images from the camera may be used to select the appropriatedevice/microphone(s) to receive the user's natural language input,and/or to select the appropriate speaker(s) to broadcast a message tothe user. In some examples and with reference to the device selectiontechniques described above, imaging data and related parameters may beincluded as a metric that is analyzed by the aggregator 182 to determinedevice selection.

In some examples, captured images of a user may be used to identifywhich device a user is facing when speaking. In some examples,indicators such as face detection may be used to identify a user. Insome examples, captured video may indicate lip movement of a user thatmay be used to associate a spoken keyword with the user. In anenvironment with multiple users, such indicators also may identify theparticular user who is addressing a device. As such, both voice andphysical recognition may be used as parameters to distinguish a userfrom among the plurality of users.

Other examples of inputs that may be used in selecting adevice/microphone and/or speaker include radar signals and lidarsignals. In some examples, signals from connected devices may indicatethat a user is interacting with the device. In one example, a user mayactivate a mobile phone via fingerprint recognition. Such an interactionmay be a strong indicator that the user is present at the location ofthe phone.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 23 schematically shows a non-limiting embodiment of a computingsystem 750 that can enact one or more of the methods and processesdescribed above. Computing system 750 is shown in simplified form.Computing system 750 may take the form of one or more personalcomputers, server computers, tablet computers, home-entertainmentcomputers, network computing devices, gaming devices, mobile computingdevices, mobile communication devices (e.g., smartphone), and/or othercomputing devices.

Computing system 750 includes a logic processor 754, volatile memory758, and a non-volatile storage device 762. Computing system 600 mayoptionally include a display subsystem 766, input subsystem 770,communication subsystem 774, and/or other components not shown in FIG.23.

Logic processor 754 includes one or more physical devices configured toexecute instructions. For example, the logic processor may be configuredto execute instructions that are part of one or more applications,programs, routines, libraries, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore components, achieve a technical effect, or otherwise arrive at adesired result.

The logic processor 754 may include one or more physical processors(hardware) configured to execute software instructions. Additionally oralternatively, the logic processor may include one or more hardwarelogic circuits or firmware devices configured to executehardware-implemented logic or firmware instructions. Processors of thelogic processor 754 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic processor optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic processor 754 may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration. In such a case, these virtualizedaspects may be run on different physical logic processors of variousdifferent machines.

Volatile memory 758 may include physical devices that include randomaccess memory. Volatile memory 758 is typically utilized by logicprocessor 754 to temporarily store information during processing ofsoftware instructions. It will be appreciated that volatile memory 758typically does not continue to store instructions when power is cut tothe volatile memory.

Non-volatile storage device 762 includes one or more physical devicesconfigured to hold instructions executable by the logic processors toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of non-volatile storage device762 may be transformed—e.g., to hold different data.

Non-volatile storage device 762 may include physical devices that areremovable and/or built-in. Non-volatile storage device 762 may includeoptical memory (CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductormemory (ROM, EPROM, EEPROM, FLASH memory, etc.), and/or magnetic memory(hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), or othermass storage device technology. Non-volatile storage device 762 mayinclude nonvolatile, dynamic, static, read/write, read-only,sequential-access, location-addressable, file-addressable, and/orcontent-addressable devices. It will be appreciated that non-volatilestorage device 762 is configured to hold instructions even when power iscut to the non-volatile storage device.

Aspects of logic processor 754, volatile memory 758, and non-volatilestorage device 762 may be integrated together into one or morehardware-logic components. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module”, “program” and “engine” may be used to describe anaspect of computing system 750 implemented to perform a particularfunction. In some cases, a module, program or engine may be instantiatedvia logic processor 754 executing instructions held by non-volatilestorage device 762, using portions of volatile memory 758. It will beunderstood that different modules, programs or engines may beinstantiated from the same application, service, code block, object,library, routine, API, function, etc. Likewise, the same module, programor engine may be instantiated by different applications, services, codeblocks, objects, routines, APIs, functions, etc. The terms modules,programs and engines encompass individual or groups of executable files,data files, libraries, drivers, scripts, database records, etc.

It will be appreciated that a “service”, as used herein, is anapplication program that may be executable across multiple usersessions. A service may be available to one or more system components,programs, and/or other services. In some implementations, a service mayrun on one or more server-computing devices.

When included, display subsystem 766 may be used to present a visualrepresentation of data held by non-volatile storage device 762. As theherein described methods and processes change the data held by thenon-volatile storage device, and thus transform the state of thenon-volatile storage device, the state of display subsystem 766 maylikewise be transformed to visually represent changes in the underlyingdata. Display subsystem 766 may include one or more display devicesutilizing virtually any type of technology. Such display devices may becombined with logic processor 754, volatile memory 758, and/ornon-volatile storage device 762 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 770 may comprise or interface with one ormore user-input devices. In some embodiments, the input subsystem maycomprise or interface with selected natural user input (NUI)componentry. Such componentry may be integrated or peripheral, and thetransduction and/or processing of input actions may be handled on- oroff-board. Example NUI componentry may include a microphone for speechand/or voice recognition; an infrared, color, stereoscopic, and/or depthcamera for machine vision and/or gesture recognition; a head tracker,eye tracker, accelerometer, and/or gyroscope for motion detection, gazedetection, and/or intent recognition; electric-field sensing componentryfor assessing brain activity; any of the sensors described with respectto the example use cases and environments discussed above; and/or anyother suitable sensor.

When included, communication subsystem 774 may be configured tocommunicatively couple computing system 750 with one or more othercomputing devices. Communication subsystem 774 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow computing system 750to send and receive data to and from other devices via a network such asthe Internet.

The following paragraphs provide additional support for the claims ofthe subject application. One aspect provides an intelligent digitalassistant system, comprising: a logic processor; and a storage deviceholding instructions executable by the logic processor to: receive audioinput comprising a speech fragment; train a first voice model with afirst sub-fragment from the speech fragment; train a second voice modelwith a second sub-fragment from the speech fragment; analyze the firstsub-fragment with the second voice model to yield a first confidencevalue; analyze the second sub-fragment with the first voice model toyield a second confidence value; and based at least on the firstconfidence value and the second confidence value, determine if a speakerof the first sub-fragment is the speaker of the second sub-fragment. Theintelligent digital assistant system may additionally or optionallyinclude, wherein the instructions are executable to, based at least ondetermining that the speaker of the first sub-fragment is the speaker ofthe second sub-fragment, utilize at least the first sub-fragment and thesecond sub-fragment to determine a user intent of the speaker. Theintelligent digital assistant system may additionally or optionallyinclude, wherein the instructions are executable to, based at least ondetermining that the speaker of the first sub-fragment is not thespeaker of the second sub-fragment, utilize at least the firstsub-fragment and forego utilizing the second sub-fragment to determine auser intent of the speaker of the first sub-fragment. The intelligentdigital assistant system may additionally or optionally include, whereinthe instructions are executable to: generate the first voice model andsecond voice model from a universal background model; and based at leaston determining that the speaker of the first sub-fragment is the speakerof the second sub-fragment, update the universal background model to anupdated universal background model using the first sub-fragment and thesecond sub-fragment. The intelligent digital assistant system mayadditionally or optionally include, wherein the instructions areexecutable to generate a third voice model from the updated universalbackground model by training the updated universal background model withanother sub-fragment of speech. The intelligent digital assistant systemmay additionally or optionally include, wherein the first sub-fragmentand the second sub-fragment have unequal temporal lengths. Theintelligent digital assistant system may additionally or optionallyinclude, wherein the instructions are executable to, based at least onthe first confidence value and the second confidence value exceeding apredetermined threshold, determine that the speaker of the firstsub-fragment is the speaker of the second sub-fragment. The intelligentdigital assistant system may additionally or optionally include, whereinthe instructions are executable to, based at least on the firstconfidence value and the second confidence value being less than orequal to a predetermined threshold, determine that the speaker of thefirst sub-fragment is not the speaker of the second sub-fragment. Theintelligent digital assistant system may additionally or optionallyinclude, wherein determining if the speaker of the first sub-fragment isthe speaker of the second sub-fragment comprises: computing an averageof the first confidence value and the second confidence value; and ifthe average exceeds a predetermined threshold, then determining that thespeaker of the first sub-fragment is the speaker of the secondsub-fragment.

Another aspect provides, at a computing device, a method for identifyinga speaker change, the method comprising: receiving audio inputcomprising a speech fragment; training a first voice model with a firstsub-fragment from the speech fragment; training a second voice modelwith a second sub-fragment from the speech fragment; analyzing the firstsub-fragment with the second voice model to yield a first confidencevalue; analyzing the second sub-fragment with the first voice model toyield a second confidence value; and based at least on the firstconfidence value and the second confidence value, determining if aspeaker of the first sub-fragment is the speaker of the secondsub-fragment. The method may additionally or optionally include, basedat least on determining that the speaker of the first sub-fragment isthe speaker of the second sub-fragment, utilizing at least the firstsub-fragment and the second sub-fragment to determine a user intent ofthe speaker. The method may additionally or optionally include, based atleast on determining that the speaker of the first sub-fragment is notthe speaker of the second sub-fragment, utilizing at least the firstsub-fragment and foregoing utilizing the second sub-fragment todetermine a user intent of the speaker of the first sub-fragment. Themethod may additionally or optionally include, generating the firstvoice model and second voice model from a universal background model;and based at least on determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment, updating theuniversal background model to an updated universal background modelusing the first sub-fragment and the second sub-fragment. The method mayadditionally or optionally include, generating a third voice model fromthe updated universal background model by training the updated universalbackground model with another sub-fragment of speech. The method mayadditionally or optionally include, wherein the first sub-fragment andthe second sub-fragment have unequal temporal lengths. The method mayadditionally or optionally include, based at least on the firstconfidence value and the second confidence value exceeding apredetermined threshold, determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment. The method mayadditionally or optionally include, based at least on the firstconfidence value and the second confidence value being less than orequal to a predetermined threshold, determining that the speaker of thefirst sub-fragment is not the speaker of the second sub-fragment. Themethod may additionally or optionally include, wherein determining ifthe speaker of the first sub-fragment is the speaker of the secondsub-fragment comprises: computing an average of the first confidencevalue and the second confidence value; and if the average exceeds apredetermined threshold, then determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment.

Another aspect provides a computing device, comprising: at least onemicrophone; a logic processor; and a storage device holding instructionsexecutable by the logic processor to: via the at least one microphone,receive audio input comprising a speech fragment; generate a firstsub-fragment and a second sub-fragment from the speech fragment; train afirst voice model with the first sub-fragment; train a second voicemodel with the second sub-fragment; analyze the first sub-fragment withthe second voice model to yield a first confidence value; analyze thesecond sub-fragment with the first voice model to yield a secondconfidence value; and based at least on the first confidence value andthe second confidence value, determine if a speaker of the firstsub-fragment is the speaker of the second sub-fragment. The computingdevice ay additionally or optionally include, wherein the instructionsare executable to, based at least on determining that the speaker of thefirst sub-fragment is the speaker of the second sub-fragment, utilize atleast the first sub-fragment and the second sub-fragment to determine auser intent of the speaker.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. An intelligent digital assistant system, comprising: a logicprocessor; and a storage device holding instructions executable by thelogic processor to: receive audio input comprising a speech fragment;train a first voice model with a first sub-fragment from the speechfragment; train a second voice model with a second sub-fragment from thespeech fragment; analyze the first sub-fragment with the second voicemodel to yield a first confidence value; analyze the second sub-fragmentwith the first voice model to yield a second confidence value; and basedat least on the first confidence value and the second confidence value,determine if a speaker of the first sub-fragment is the speaker of thesecond sub-fragment.
 2. The intelligent digital assistant system ofclaim of 1, wherein the instructions are executable to, based at leaston determining that the speaker of the first sub-fragment is the speakerof the second sub-fragment, utilize at least the first sub-fragment andthe second sub-fragment to determine a user intent of the speaker. 3.The intelligent digital assistant system of claim of 1, wherein theinstructions are executable to, based at least on determining that thespeaker of the first sub-fragment is not the speaker of the secondsub-fragment, utilize at least the first sub-fragment and foregoutilizing the second sub-fragment to determine a user intent of thespeaker of the first sub-fragment.
 4. The intelligent digital assistantsystem of claim 1, wherein the instructions are executable to: generatethe first voice model and second voice model from a universal backgroundmodel; and based at least on determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment, update theuniversal background model to an updated universal background modelusing the first sub-fragment and the second sub-fragment.
 5. Theintelligent digital assistant system of claim of 4, wherein theinstructions are executable to generate a third voice model from theupdated universal background model by training the updated universalbackground model with another sub-fragment of speech.
 6. The intelligentdigital assistant system of claim 1, wherein the first sub-fragment andthe second sub-fragment have unequal temporal lengths.
 7. Theintelligent digital assistant system of claim 1, wherein theinstructions are executable to, based at least on the first confidencevalue and the second confidence value exceeding a predeterminedthreshold, determine that the speaker of the first sub-fragment is thespeaker of the second sub-fragment.
 8. The intelligent digital assistantsystem of claim 1, wherein the instructions are executable to, based atleast on the first confidence value and the second confidence valuebeing less than or equal to a predetermined threshold, determine thatthe speaker of the first sub-fragment is not the speaker of the secondsub-fragment.
 9. The intelligent digital assistant system of claim 1,wherein determining if the speaker of the first sub-fragment is thespeaker of the second sub-fragment comprises: computing an average ofthe first confidence value and the second confidence value; and if theaverage exceeds a predetermined threshold, then determining that thespeaker of the first sub-fragment is the speaker of the secondsub-fragment.
 10. At a computing device, a method for identifying aspeaker change, the method comprising: receiving audio input comprisinga speech fragment; training a first voice model with a firstsub-fragment from the speech fragment; training a second voice modelwith a second sub-fragment from the speech fragment; analyzing the firstsub-fragment with the second voice model to yield a first confidencevalue; analyzing the second sub-fragment with the first voice model toyield a second confidence value; and based at least on the firstconfidence value and the second confidence value, determining if aspeaker of the first sub-fragment is the speaker of the secondsub-fragment.
 11. The method of claim 10, further comprising, based atleast on determining that the speaker of the first sub-fragment is thespeaker of the second sub-fragment, utilizing at least the firstsub-fragment and the second sub-fragment to determine a user intent ofthe speaker.
 12. The method of claim 10, further comprising, based atleast on determining that the speaker of the first sub-fragment is notthe speaker of the second sub-fragment, utilizing at least the firstsub-fragment and foregoing utilizing the second sub-fragment todetermine a user intent of the speaker of the first sub-fragment. 13.The method of claim 10, further comprising: generating the first voicemodel and second voice model from a universal background model; andbased at least on determining that the speaker of the first sub-fragmentis the speaker of the second sub-fragment, updating the universalbackground model to an updated universal background model using thefirst sub-fragment and the second sub-fragment.
 14. The method of claim13, further comprising generating a third voice model from the updateduniversal background model by training the updated universal backgroundmodel with another sub-fragment of speech.
 15. The method of claim 10,wherein the first sub-fragment and the second sub-fragment have unequaltemporal lengths.
 16. The method of claim 10, further comprising, basedat least on the first confidence value and the second confidence valueexceeding a predetermined threshold, determining that the speaker of thefirst sub-fragment is the speaker of the second sub-fragment.
 17. Themethod of claim 10, further comprising, based at least on the firstconfidence value and the second confidence value being less than orequal to a predetermined threshold, determining that the speaker of thefirst sub-fragment is not the speaker of the second sub-fragment. 18.The method of claim 10, wherein determining if the speaker of the firstsub-fragment is the speaker of the second sub-fragment comprises:computing an average of the first confidence value and the secondconfidence value; and if the average exceeds a predetermined threshold,then determining that the speaker of the first sub-fragment is thespeaker of the second sub-fragment.
 19. A computing device, comprising:at least one microphone; a logic processor; and a storage device holdinginstructions executable by the logic processor to: via the at least onemicrophone, receive audio input comprising a speech fragment; generate afirst sub-fragment and a second sub-fragment from the speech fragment;train a first voice model with the first sub-fragment; train a secondvoice model with the second sub-fragment; analyze the first sub-fragmentwith the second voice model to yield a first confidence value; analyzethe second sub-fragment with the first voice model to yield a secondconfidence value; and based at least on the first confidence value andthe second confidence value, determine if a speaker of the firstsub-fragment is the speaker of the second sub-fragment.
 20. Thecomputing device of claim 19, wherein the instructions are executableto, based at least on determining that the speaker of the firstsub-fragment is the speaker of the second sub-fragment, utilize at leastthe first sub-fragment and the second sub-fragment to determine a userintent of the speaker.